Oct 30 18:36:56.656100 ip-10-0-129-31 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:36:57.119546 ip-10-0-129-31 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:57.119546 ip-10-0-129-31 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:36:57.119546 ip-10-0-129-31 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:57.119546 ip-10-0-129-31 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:57.123179 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123061    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.123179 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123180    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123183    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123187    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123189    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123192    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123195    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123197    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123202    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123206    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123209    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123212    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123215    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123218    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123221    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123227    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123230    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123232    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123235    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123237    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.123246 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123240    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123242    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123245    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123248    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123250    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123253    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123256    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123259    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123262    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123264    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123268    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123271    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123274    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123277    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123279    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123282    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123284    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123287    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123289    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.123692 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123292    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123294    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123297    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123300    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123302    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123305    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123307    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123310    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123312    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123314    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123317    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123319    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123322    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123325    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123328    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123331    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123333    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123335    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123338    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123340    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.124184 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123343    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123345    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123348    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123350    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123361    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123364    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123366    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123369    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123373    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123375    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123378    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123380    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123383    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123385    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123388    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123390    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123394    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123397    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123399    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123402    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.124643 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123404    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123407    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123409    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123412    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123414    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123417    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123420    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123422    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123425    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123428    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123431    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123433    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123436    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123438    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123441    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123443    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123446    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123448    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123451    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123453    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.125156 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123456    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123458    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123461    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123464    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123467    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123469    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.123472    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124233    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124240    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124244    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124246    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124250    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124252    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124255    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124257    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124260    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124263    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124266    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124269    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.125619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124272    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124274    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124277    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124280    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124282    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124285    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124287    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124290    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124293    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124295    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124298    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124300    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124303    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124305    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124308    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124310    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124313    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124315    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124318    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.126113 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124320    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124323    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124325    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124327    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124330    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124333    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124335    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124338    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124340    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124343    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124345    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124348    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124352    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124354    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124357    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124359    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124362    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124364    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124367    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.126565 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124369    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124371    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124374    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124376    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124379    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124381    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124383    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124386    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124388    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124391    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124393    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124395    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124398    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124400    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124403    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124405    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124407    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124410    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124413    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.127070 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124415    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124418    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124420    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124422    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124425    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124428    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124432    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124434    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124437    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124439    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124441    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124444    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124447    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124449    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124451    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124454    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124456    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124459    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124461    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124465    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.127520 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124469    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124474    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124477    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124480    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124483    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124485    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124488    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124491    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124493    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124496    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124498    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124501    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124505    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124509    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124512    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124515    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124518    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.124995    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125001    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125003    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.128002 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125006    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125009    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125012    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125015    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125017    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125020    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125022    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125025    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125027    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125030    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125032    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125036    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125038    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125040    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125043    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125045    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125048    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125051    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125053    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125056    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.128499 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125058    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125060    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125063    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125067    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125070    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125073    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125075    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125078    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125080    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125083    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125085    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125087    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125090    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125092    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125095    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125097    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125100    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125102    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125104    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125107    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.128984 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125109    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125112    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125114    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125119    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125122    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125125    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125128    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125131    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125134    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125137    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125140    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125143    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125145    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125148    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125150    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125156    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125159    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125162    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125165    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.129450 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125168    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125173    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125176    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125180    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125182    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125185    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125188    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125191    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125193    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125196    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125198    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125201    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125203    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125206    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125208    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125211    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125213    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125216    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125218    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.129919 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125221    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125223    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125226    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125228    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125231    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125233    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125236    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125239    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125241    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125244    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125246    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125249    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125252    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125254    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125257    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125259    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125262    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125264    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125268    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.130362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125270    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125272    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125275    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125277    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125279    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.125282    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.127997    2578 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128007    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128012    2578 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128017    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128021    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128024    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128028    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128032    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128035    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128038    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128041    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128044    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128047    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128050    2578 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128053    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128056    2578 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128059    2578 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:36:57.130805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128062    2578 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128067    2578 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128069    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128073    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128076    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128079    2578 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128083    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128086    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128089    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128092    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128095    2578 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128098    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128101    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128104    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128107    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128111    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128114    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128117    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128120    2578 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128123    2578 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128126    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128129    2578 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128132    2578 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128135    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:36:57.131332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128138    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128141    2578 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128146    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128148    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128151    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128155    2578 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128158    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128161    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128164    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128167    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128170    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128173    2578 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128176    2578 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128180    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128183    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128187    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128190    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128193    2578 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128196    2578 flags.go:64] FLAG: --help="false"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128199    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128202    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128205    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128208    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:36:57.131974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128211    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128214    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128217    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128220    2578 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128223    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128226    2578 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128228    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128232    2578 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128234    2578 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128237    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128240    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128243    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128246    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128249    2578 flags.go:64] FLAG: --lock-file=""
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128251    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128254    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128257    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128263    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128266    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128269    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128272    2578 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128274    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128278    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128281    2578 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128284    2578 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:36:57.132519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128288    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128291    2578 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128295    2578 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128298    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128301    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128303    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128306    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128309    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128312    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128314    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128322    2578 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128325    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128327    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128330    2578 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128333    2578 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128336    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128339    2578 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128342    2578 flags.go:64] FLAG: --port="10250"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128345    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128348    2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ff983347bfafa4b2"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128351    2578 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128354    2578 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128357    2578 flags.go:64] FLAG: --register-node="true"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128360    2578 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128364    2578 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128367    2578 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:36:57.133128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128370    2578 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128372    2578 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128376    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128379    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128383    2578 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128386    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128389    2578 flags.go:64] FLAG: --runonce="false"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128392    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128395    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128398    2578 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128401    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128403    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128406    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128409    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128412    2578 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128415    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128418    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128421    2578 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128423    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128426    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128429    2578 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128432    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128437    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128440    2578 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128443    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:36:57.133735 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128448    2578 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128450    2578 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128453    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128456    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128459    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128462    2578 flags.go:64] FLAG: --v="2"
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128466    2578 flags.go:64] FLAG: --version="false"
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128470    2578 flags.go:64] FLAG: --vmodule=""
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128474    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128477    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128608    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128611    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128615    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128618    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128621    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128624    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128626    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128629    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128631    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128633    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128637    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.134380 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128640    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128644    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128647    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128650    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128652    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128655    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128658    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128660    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128663    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128665    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128668    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128671    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128673    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128675    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128678    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128681    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128684    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128686    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128688    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.134895 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128691    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128693    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128696    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128698    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128701    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128704    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128707    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128709    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128712    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128714    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128717    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128720    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128722    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128724    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128727    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128729    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128732    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128734    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128736    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128739    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.135346 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128742    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128744    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128747    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128749    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128751    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128754    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128756    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128759    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128761    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128763    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128766    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128768    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128770    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128773    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128776    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128780    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128784    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128788    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128794    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128814    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128818    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.135876 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128823    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128826    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128830    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128833    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128836    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128838    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128841    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128843    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128846    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128848    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128851    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128857    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128860    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128862    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128865    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128868    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128870    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128873    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128875    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.136362 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128878    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128881    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128886    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128890    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128909    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128913    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128915    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128918    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128920    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128923    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128926    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128929    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128932    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128934    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128936    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.136813 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.128939    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.128968    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129012    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129018    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129022    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129025    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129028    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129030    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129033    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129037    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129041    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129044    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129046    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129048    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129051    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.137174 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129056    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129060    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129065    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129069    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129073    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129076    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129079    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129082    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129084    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129087    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129089    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129092    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129094    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129097    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129100    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129103    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129105    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129108    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129110    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.137526 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129113    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129115    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129118    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129120    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129123    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129125    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129128    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129131    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129137    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129141    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129146    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129150    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129155    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129158    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129162    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129166    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129168    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129171    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129173    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.137991 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129176    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129179    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129181    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129184    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129186    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129188    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129191    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129193    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129196    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129199    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129202    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129205    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129208    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129210    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129212    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129215    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129220    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129225    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129229    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129234    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.138480 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129238    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129241    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129244    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129247    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129249    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129251    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129254    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129256    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129259    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129261    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129264    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129266    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129269    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129271    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129273    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129276    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129278    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129281    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129283    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129286    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.139043 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129289    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129291    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129294    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129297    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129302    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129306    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129310    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129315    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129318    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129320    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129323    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129325    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129329    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129333    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.129336    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.136887    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.136902    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.137545    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.137551    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137618    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.139995 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137623    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137627    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137630    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137633    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137636    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137639    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137642    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137645    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137647    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137650    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137652    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137655    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137658    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137660    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137663    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137666    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137669    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137671    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137673    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137676    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.140469 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137678    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137681    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137684    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137686    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137689    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137692    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137696    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137700    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137703    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137706    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137709    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137712    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137714    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137717    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137719    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137721    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137724    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137726    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137729    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.140952 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137731    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137734    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137736    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137738    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137741    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137743    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137745    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137748    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137751    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137753    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137756    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137758    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137761    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137763    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137766    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137768    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137771    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137773    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137775    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.141386 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137778    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137781    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137783    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137786    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137788    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137790    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137793    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137814    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137817    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137819    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137822    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137824    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137827    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137829    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137832    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137834    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137837    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137839    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137842    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137844    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.141881 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137847    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137849    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137852    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137855    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137859    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137863    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137866    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137869    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137871    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137874    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137876    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137879    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137881    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137884    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137887    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137889    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137892    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137894    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137897    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.142377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137899    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137902    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137904    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137907    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137909    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137911    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137914    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.137916    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.137944    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138069    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138073    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138076    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138078    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138081    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138084    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138086    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.142841 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138089    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138091    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138095    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138099    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138102    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138105    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138107    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138110    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138112    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138115    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138117    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138120    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138123    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138125    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138127    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138130    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138132    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138134    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138137    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138139    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.143229 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138142    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138144    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138147    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138149    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138151    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138154    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138156    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138158    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138161    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138163    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138166    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138168    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138170    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138173    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138176    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138178    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138181    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138183    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138185    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138188    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.143731 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138190    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138193    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138195    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138198    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138200    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138202    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138205    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138207    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138210    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138212    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138215    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138217    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138220    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138222    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138224    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138227    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138229    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138231    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138234    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138236    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.144262 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138239    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138241    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138243    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138246    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138248    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138251    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138286    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138323    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138326    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138329    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138332    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.138335    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139466    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139480    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139485    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139490    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139495    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139501    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139529    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.144728 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139550    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139667    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139676    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139682    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139686    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139689    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139692    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139696    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139700    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139702    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139705    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139707    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139710    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139715    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139720    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139724    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139727    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139729    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139732    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.145180 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139735    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.139772    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139833    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139838    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139841    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139844    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139847    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139851    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139853    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139856    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139858    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139861    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139863    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139866    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139868    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139871    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.145619 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139874    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139876    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139879    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139883    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139885    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139888    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139890    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139893    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139895    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139898    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139901    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139903    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139905    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139908    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139911    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139913    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139916    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139918    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139921    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.146036 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139924    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139927    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139929    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139931    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139935    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139939    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139943    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139946    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139949    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139951    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139954    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139956    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139958    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139961    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139963    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139966    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139968    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139971    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139973    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.146473 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139976    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139979    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139983    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139987    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139990    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139993    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139995    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.139998    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140000    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140003    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140006    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140009    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140012    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140015    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140017    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140020    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140022    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140026    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140029    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140031    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.146915 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140034    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140037    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140039    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140042    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140045    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140047    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140050    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140052    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140055    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140057    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140061    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140063    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140066    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140068    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140071    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140073    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140076    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140078    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140081    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140083    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.147531 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140086    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140088    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140091    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140096    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140098    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140101    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140103    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140106    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140108    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140112    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140114    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140117    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140120    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:57.140124    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.140181    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.142143    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.143280    2578 server.go:1014] "Starting client certificate rotation"
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.143386    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:57.148029 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.143424    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:57.148477 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.145854    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:57.148477 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.147015    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:57.158876 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.158857    2578 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:36:57.162042 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.162029    2578 log.go:25] "Validated CRI v1 image API"
Oct 30 18:36:57.163660 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.163629    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:36:57.169224 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.169207    2578 fs.go:135] Filesystem UUIDs: map[0411990e-163a-415d-87eb-de467208efee:/dev/nvme0n1p3 67ce03ed-9ba8-4c2d-995e-c834f89d14c0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 18:36:57.169275 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.169224    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:36:57.174646 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.174543    2578 manager.go:217] Machine: {Timestamp:2026-10-30 18:36:57.172692486 +0000 UTC m=+0.388426288 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103609 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a0beaa8721ca71972feb5eae7b9ed SystemUUID:ec2a0bea-a872-1ca7-1972-feb5eae7b9ed BootID:4cab329c-1a0c-4033-a1ff-40f2e26f657f Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ca:56:50:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ca:56:50:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:49:dc:c6:21:0a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:36:57.174646 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.174646    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:36:57.174740 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.174710    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:36:57.177460 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.177440    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:36:57.177580 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.177462    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-31.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:36:57.177621 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.177582    2578 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:36:57.177621 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.177590    2578 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:36:57.177673 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.177622    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:57.177673 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.177640    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:57.178470 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.178461    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:36:57.182262 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.182247    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:36:57.182368 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.182359    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:36:57.183109 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.183100    2578 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:36:57.183139 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.183112    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:36:57.183139 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.183122    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:36:57.183139 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.183130    2578 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:36:57.183139 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.183137    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:36:57.183423 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.183412    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:57.187675 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.187655    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:36:57.192785 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.192772    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:57.194045 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.194033    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:36:57.194088 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.194064    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:36:57.194222 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.194211    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:36:57.194250 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.194231    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:36:57.194250 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.194237    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:36:57.194250 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.194242    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:36:57.194250 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.194247    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:36:57.194345 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.194253    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:36:57.195157 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.195149    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:36:57.195225 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.195158    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:36:57.195225 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.195165    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:36:57.195225 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.195171    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:36:57.195225 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.195179    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:36:57.195225 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.195187    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:36:57.195906 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.195895    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:36:57.195939 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.195908    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:36:57.200307 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.200294    2578 server.go:1263] "Started kubelet"
Oct 30 18:36:57.200397 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.200362    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:36:57.200449 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.200394    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:36:57.200483 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.200463    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:36:57.201031 ip-10-0-129-31 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:36:57.201692 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.201671    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:36:57.203679 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.203665    2578 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:36:57.207078 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.207057    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:57.207582 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.207563    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:36:57.207735 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.207712    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:36:57.207858 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.207835    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:36:57.208192 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208175    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:36:57.208383 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208292    2578 factory.go:55] Registering systemd factory
Oct 30 18:36:57.208469 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208391    2578 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:36:57.208469 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208310    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:36:57.208469 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208439    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:36:57.208587 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208539    2578 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:36:57.208587 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208547    2578 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:36:57.208587 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208585    2578 factory.go:153] Registering CRI-O factory
Oct 30 18:36:57.208675 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208597    2578 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:36:57.208675 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208636    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:36:57.208675 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208649    2578 factory.go:103] Registering Raw factory
Oct 30 18:36:57.208675 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.208658    2578 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:36:57.209755 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.209739    2578 manager.go:319] Starting recovery of all containers
Oct 30 18:36:57.219501 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.219331    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:36:57.221330 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.221316    2578 manager.go:324] Recovery completed
Oct 30 18:36:57.222462 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.222437    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 18:36:57.225290 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.225276    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:57.226851 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.226826    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:57.226924 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.226868    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:57.226924 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.226885    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:57.227459 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.227446    2578 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:36:57.227459 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.227458    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:36:57.227567 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.227473    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:36:57.227567 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.227482    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:57.228133 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.228118    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:57.228182 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.228149    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:57.228182 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.228162    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:57.229540 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.229529    2578 policy_none.go:50] "Start"
Oct 30 18:36:57.229581 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.229545    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:36:57.229581 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.229558    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:36:57.231424 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.231414    2578 policy_none.go:44] "Start"
Oct 30 18:36:57.270350 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.270326    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:36:57.272847 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.272832    2578 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:36:57.272908 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.272859    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:36:57.272908 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.272868    2578 server.go:87] "Starting device plugin registration server"
Oct 30 18:36:57.273098 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.273085    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:36:57.273184 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.273101    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:36:57.273184 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.273181    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:36:57.273261 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.273245    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:36:57.273261 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.273253    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:36:57.273731 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.273713    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:36:57.273792 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.273764    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:36:57.339729 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.339678    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:36:57.339729 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.339699    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:36:57.339729 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.339714    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:36:57.341231 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.339744    2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:36:57.344888 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.344864    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:36:57.344888 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.344871    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:36:57.353859 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.344852    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e3628901965d19  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.200262425 +0000 UTC m=+0.415996227,LastTimestamp:2026-10-30 18:36:57.200262425 +0000 UTC m=+0.415996227,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:57.373643 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.373627    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:57.374278 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.374266    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:57.374331 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.374293    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:57.374331 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.374308    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:57.374393 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.374341    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.398166 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.398148    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:36:57.398488 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.398471    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:36:57.398532 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.398511    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:36:57.434963 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.434944    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:36:57.435024 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.434981    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pdzj2"
Oct 30 18:36:57.440410 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.440395    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"]
Oct 30 18:36:57.440461 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.440448    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:57.441239 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.441224    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:57.441313 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.441252    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:57.441313 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.441266    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:57.442989 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.442978    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:57.443138 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.443123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.443170 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.443153    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:57.443607 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.443590    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:57.443607 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.443618    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:57.443745 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.443635    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:57.443745 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.443696    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:57.443745 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.443723    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:57.443745 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.443737    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:57.445564 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.445548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.445646 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.445578    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:57.446366 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.446350    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:57.446437 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.446377    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:57.446437 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.446388    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:57.469511 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.469492    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.471523 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.471504    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.488117 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.488094    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 18:36:57.488257 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.488180    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226850888 +0000 UTC m=+0.442584696,LastTimestamp:2026-10-30 18:36:57.226850888 +0000 UTC m=+0.442584696,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:57.510335 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.510314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.510418 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.510337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.510418 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.510355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d71fab1c4baec8ab7a9648db13d15f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"38d71fab1c4baec8ab7a9648db13d15f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.559197 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.559094    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226875266 +0000 UTC m=+0.442609073,LastTimestamp:2026-10-30 18:36:57.226875266 +0000 UTC m=+0.442609073,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:57.559358 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.559326    2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.611159 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.611115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.611159 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.611142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.611292 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.611159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d71fab1c4baec8ab7a9648db13d15f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"38d71fab1c4baec8ab7a9648db13d15f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.611292 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.611193    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d71fab1c4baec8ab7a9648db13d15f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"38d71fab1c4baec8ab7a9648db13d15f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.611292 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.611202    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.611292 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.611232    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.644736 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.644666    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226891047 +0000 UTC m=+0.442624855,LastTimestamp:2026-10-30 18:36:57.226891047 +0000 UTC m=+0.442624855,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:57.644989 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.644951    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Oct 30 18:36:57.708512 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.708446    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226850888 +0000 UTC m=+0.442584696,LastTimestamp:2026-10-30 18:36:57.228132865 +0000 UTC m=+0.443866666,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:57.759919 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.759901    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:57.761861 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.761846    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:57.761926 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.761872    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:57.761926 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.761883    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:57.761926 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.761905    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.773026 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.773004    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.775545 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:57.775531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.789542 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.789485    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226875266 +0000 UTC m=+0.442609073,LastTimestamp:2026-10-30 18:36:57.228156317 +0000 UTC m=+0.443890119,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:57.852462 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.852385    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226891047 +0000 UTC m=+0.442624855,LastTimestamp:2026-10-30 18:36:57.228165664 +0000 UTC m=+0.443899465,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:57.939836 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.939776    2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:57.939956 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:57.939900    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e3628905e95066  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.272807526 +0000 UTC m=+0.488541327,LastTimestamp:2026-10-30 18:36:57.272807526 +0000 UTC m=+0.488541327,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.009539 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.009481    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289060a7bfd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.274981373 +0000 UTC m=+0.490715165,LastTimestamp:2026-10-30 18:36:57.274981373 +0000 UTC m=+0.490715165,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.091032 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.090971    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226850888 +0000 UTC m=+0.442584696,LastTimestamp:2026-10-30 18:36:57.374278847 +0000 UTC m=+0.590012649,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.169382 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.169355    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms"
Oct 30 18:36:58.169787 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.169361    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226875266 +0000 UTC m=+0.442609073,LastTimestamp:2026-10-30 18:36:57.374299696 +0000 UTC m=+0.590033503,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.238546 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.238469    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226891047 +0000 UTC m=+0.442624855,LastTimestamp:2026-10-30 18:36:57.374314766 +0000 UTC m=+0.590048570,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.314760 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.314697    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226850888 +0000 UTC m=+0.442584696,LastTimestamp:2026-10-30 18:36:57.441238088 +0000 UTC m=+0.656971894,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.339966 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:58.339952    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:58.340728 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:58.340706    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:58.340855 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:58.340743    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:58.340855 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:58.340759    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:58.340855 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:58.340787    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:58.398925 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.398814    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226875266 +0000 UTC m=+0.442609073,LastTimestamp:2026-10-30 18:36:57.441259914 +0000 UTC m=+0.656993716,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.399077 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:58.399060    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:36:58.462565 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.462508    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226891047 +0000 UTC m=+0.442624855,LastTimestamp:2026-10-30 18:36:57.441269532 +0000 UTC m=+0.657003333,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.462664 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.462597    2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:36:58.462819 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.462780    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:36:58.465239 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:58.465224    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:58.466037 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:58.466018    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:58.466134 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:58.466054    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:58.466134 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:58.466068    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:58.547972 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.547917    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:36:58.547972 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.547885    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226850888 +0000 UTC m=+0.442584696,LastTimestamp:2026-10-30 18:36:57.443607326 +0000 UTC m=+0.659341134,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.609200 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.609138    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226875266 +0000 UTC m=+0.442609073,LastTimestamp:2026-10-30 18:36:57.443626354 +0000 UTC m=+0.659360159,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.693330 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.693268    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226891047 +0000 UTC m=+0.442624855,LastTimestamp:2026-10-30 18:36:57.44364169 +0000 UTC m=+0.659375498,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.773713 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.773608    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226850888 +0000 UTC m=+0.442584696,LastTimestamp:2026-10-30 18:36:57.443711296 +0000 UTC m=+0.659445109,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.844096 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:58.843616    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d71fab1c4baec8ab7a9648db13d15f.slice/crio-18b2914f029dcd2d7d184b647181af4988040700ee0d3989d5daa4996aa5b4da WatchSource:0}: Error finding container 18b2914f029dcd2d7d184b647181af4988040700ee0d3989d5daa4996aa5b4da: Status 404 returned error can't find the container with id 18b2914f029dcd2d7d184b647181af4988040700ee0d3989d5daa4996aa5b4da
Oct 30 18:36:58.844339 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:36:58.844319    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c96e16bc130f55ab259bba0b1cc606.slice/crio-436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685 WatchSource:0}: Error finding container 436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685: Status 404 returned error can't find the container with id 436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685
Oct 30 18:36:58.848378 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:36:58.848364    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:36:58.854630 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.854568    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226875266 +0000 UTC m=+0.442609073,LastTimestamp:2026-10-30 18:36:57.443729345 +0000 UTC m=+0.659463148,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.916602 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.916537    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226891047 +0000 UTC m=+0.442624855,LastTimestamp:2026-10-30 18:36:57.443742479 +0000 UTC m=+0.659476283,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:36:58.916790 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.916770    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:36:58.916938 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:36:58.916919    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 18:37:00.715608 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:00.715516    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226850888 +0000 UTC m=+0.442584696,LastTimestamp:2026-10-30 18:36:57.446365185 +0000 UTC m=+0.662098986,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:00.780865 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:00.780841    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s"
Oct 30 18:37:00.780948 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:00.780863    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c7182  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226875266 +0000 UTC m=+0.442609073,LastTimestamp:2026-10-30 18:36:57.446381884 +0000 UTC m=+0.662115686,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:00.873658 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:00.873588    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032caf27  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226891047 +0000 UTC m=+0.442624855,LastTimestamp:2026-10-30 18:36:57.446391626 +0000 UTC m=+0.662125428,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:00.924385 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:00.924318    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-31.us-west-2.compute.internal.18e36289032c1248  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-31.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-31.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:57.226850888 +0000 UTC m=+0.442584696,LastTimestamp:2026-10-30 18:36:57.761860431 +0000 UTC m=+0.977594236,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:00.978417 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:00.978399    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:00.979406 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:00.979389    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:00.979475 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:00.979420    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:00.979475 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:00.979436    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:00.979475 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:00.979463    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:01.036204 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:01.036123    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal.18e3628963dab183  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal,UID:38d71fab1c4baec8ab7a9648db13d15f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\",Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:58.848907651 +0000 UTC m=+2.064641440,LastTimestamp:2026-10-30 18:36:58.848907651 +0000 UTC m=+2.064641440,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:01.059139 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:01.059095    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"38d71fab1c4baec8ab7a9648db13d15f","Type":"ContainerStarted","Data":"18b2914f029dcd2d7d184b647181af4988040700ee0d3989d5daa4996aa5b4da"}
Oct 30 18:37:01.060180 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:01.060160    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"e1c96e16bc130f55ab259bba0b1cc606","Type":"ContainerStarted","Data":"436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685"}
Oct 30 18:37:01.127784 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:01.127734    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:01.127784 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:01.127741    2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:01.127923 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:01.127860    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal.18e3628963f1f663  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal,UID:e1c96e16bc130f55ab259bba0b1cc606,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\",Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:58.850432611 +0000 UTC m=+2.066166401,LastTimestamp:2026-10-30 18:36:58.850432611 +0000 UTC m=+2.066166401,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:02.109305 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:02.109273    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:02.421638 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:02.421566    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:02.426180 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:02.426143    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:02.426310 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:02.426187    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:02.426310 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:02.426206    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:02.452403 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:02.452376    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s"
Oct 30 18:37:02.529633 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:02.529593    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:37:02.718433 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:02.718361    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal.18e3628a475a8ded  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal,UID:38d71fab1c4baec8ab7a9648db13d15f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\" in 2.101s (2.101s including waiting). Image size: 294084393 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:02.665715181 +0000 UTC m=+4.165718777,LastTimestamp:2026-10-30 18:37:02.665715181 +0000 UTC m=+4.165718777,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:02.728634 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:02.728614    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:02.729363 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:02.729346    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:02.729430 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:02.729382    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:02.729430 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:02.729395    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:02.729430 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:02.729420    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:02.880083 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:02.879978    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal.18e3628a4c034057  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal,UID:38d71fab1c4baec8ab7a9648db13d15f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:02.743879767 +0000 UTC m=+4.243883360,LastTimestamp:2026-10-30 18:37:02.743879767 +0000 UTC m=+4.243883360,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:03.026105 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:03.026047    2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:03.026206 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:03.026081    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal.18e3628a4c70b1ad  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal,UID:38d71fab1c4baec8ab7a9648db13d15f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:02.751052205 +0000 UTC m=+4.251056020,LastTimestamp:2026-10-30 18:37:02.751052205 +0000 UTC m=+4.251056020,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:03.064202 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:03.064178    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"38d71fab1c4baec8ab7a9648db13d15f","Type":"ContainerStarted","Data":"fef671a98bae65212702d6bb17f9dcf96ca27e78f3495f1d606023a31fb3ed96"}
Oct 30 18:37:03.064307 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:03.064237    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:03.064902 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:03.064887    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:03.064984 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:03.064920    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:03.064984 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:03.064935    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:03.065124 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:03.065111    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:03.177683 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:03.177661    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:03.337299 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:03.337269    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:37:03.494740 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:03.494707    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:37:03.783580 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:03.783476    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal.18e3628a8155b75e  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal,UID:e1c96e16bc130f55ab259bba0b1cc606,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\" in 3.072s (3.072s including waiting). Image size: 272339155 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:03.638476638 +0000 UTC m=+5.138480230,LastTimestamp:2026-10-30 18:37:03.638476638 +0000 UTC m=+5.138480230,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:03.943373 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:03.943246    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal.18e3628a8e0f01f4  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal,UID:e1c96e16bc130f55ab259bba0b1cc606,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:03.851946484 +0000 UTC m=+5.351950099,LastTimestamp:2026-10-30 18:37:03.851946484 +0000 UTC m=+5.351950099,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:04.019280 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:04.019250    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 18:37:04.019367 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:04.019256    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal.18e3628a8e90b933  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal,UID:e1c96e16bc130f55ab259bba0b1cc606,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:03.860447539 +0000 UTC m=+5.360451136,LastTimestamp:2026-10-30 18:37:03.860447539 +0000 UTC m=+5.360451136,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:04.067477 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:04.067445    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"e1c96e16bc130f55ab259bba0b1cc606","Type":"ContainerStarted","Data":"abdbf86c78b8cca0528108aebac6f1ba860cc40bd481cd465bd4fccb7f131697"}
Oct 30 18:37:04.067573 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:04.067478    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:04.067573 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:04.067486    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:04.068442 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:04.068417    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:04.068442 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:04.068426    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:04.068587 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:04.068454    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:04.068587 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:04.068467    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:04.068587 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:04.068455    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:04.068587 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:04.068574    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:04.068714 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:04.068673    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:04.068768 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:04.068752    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:04.112272 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:04.112251    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:05.068775 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:05.068750    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:05.069499 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:05.069483    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:05.069594 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:05.069513    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:05.069594 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:05.069539    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:05.069754 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:05.069741    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:05.208132 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:05.208108    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:05.810034 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:05.810007    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s"
Oct 30 18:37:06.205717 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:06.205691    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:06.227039 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:06.227014    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:06.227945 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:06.227916    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:06.228055 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:06.227957    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:06.228055 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:06.227969    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:06.228055 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:06.227999    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:06.533104 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:06.533049    2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:06.653478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:06.653447    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:06.654437 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:06.654421    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:06.654550 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:06.654455    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:06.654550 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:06.654469    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:07.220247 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:07.220224    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:08.222795 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:08.222770    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:08.938541 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:08.938498    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:37:08.938541 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:08.938508    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:37:08.989799 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:08.989774    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:09.108812 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:09.108780    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:09.510721 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:09.510695    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:37:10.194972 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:10.194943    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:10.361347 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:10.361315    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 18:37:11.219957 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:11.219929    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:12.209939 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:12.209907    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:12.367159 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:12.367114    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Oct 30 18:37:12.934025 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:12.933998    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:12.934996 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:12.934973    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:12.935070 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:12.935011    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:12.935070 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:12.935022    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:12.935070 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:12.935051    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:13.225750 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:13.225682    2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:13.390643 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:13.390615    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:14.219861 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:14.219835    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:15.084417 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:15.084386    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_e1c96e16bc130f55ab259bba0b1cc606/setup/0.log"
Oct 30 18:37:15.084843 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:15.084435    2578 generic.go:356] "Generic (PLEG): container finished" podID="e1c96e16bc130f55ab259bba0b1cc606" containerID="abdbf86c78b8cca0528108aebac6f1ba860cc40bd481cd465bd4fccb7f131697" exitCode=1
Oct 30 18:37:15.084843 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:15.084476    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"e1c96e16bc130f55ab259bba0b1cc606","Type":"ContainerDied","Data":"abdbf86c78b8cca0528108aebac6f1ba860cc40bd481cd465bd4fccb7f131697"}
Oct 30 18:37:15.084843 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:15.084705    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:15.086090 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:15.086062    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:15.086225 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:15.086100    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:15.086225 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:15.086116    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:15.086344 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:15.086329    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:15.205138 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:15.205112    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:15.205230 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:15.205120    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal.18e3628d2be7aa9f  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal,UID:e1c96e16bc130f55ab259bba0b1cc606,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\" already present on machine and can be accessed by the pod,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:15.090102943 +0000 UTC m=+16.590106554,LastTimestamp:2026-10-30 18:37:15.090102943 +0000 UTC m=+16.590106554,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:15.364243 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:15.364168    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal.18e3628a8e0f01f4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal.18e3628a8e0f01f4  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal,UID:e1c96e16bc130f55ab259bba0b1cc606,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:03.851946484 +0000 UTC m=+5.351950099,LastTimestamp:2026-10-30 18:37:15.166520901 +0000 UTC m=+16.666524494,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:15.519759 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:15.519654    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal.18e3628a8e90b933\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal.18e3628a8e90b933  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal,UID:e1c96e16bc130f55ab259bba0b1cc606,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-129-31.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:37:03.860447539 +0000 UTC m=+5.360451136,LastTimestamp:2026-10-30 18:37:15.174692816 +0000 UTC m=+16.674696413,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-31.us-west-2.compute.internal,}"
Oct 30 18:37:16.088340 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:16.088315    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_e1c96e16bc130f55ab259bba0b1cc606/setup/0.log"
Oct 30 18:37:16.088744 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:16.088359    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"e1c96e16bc130f55ab259bba0b1cc606","Type":"ContainerStarted","Data":"324ec2c193e25b0b6e5c1614b81a8c2aaa1409f60467d2d5a80863aabbef354f"}
Oct 30 18:37:16.088744 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:16.088478    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:16.089292 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:16.089272    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:16.089369 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:16.089314    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:16.089369 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:16.089329    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:16.089604 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:16.089586    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:16.221524 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:16.221493    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:16.344187 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:16.344126    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:16.345095 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:16.345072    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:16.345183 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:16.345108    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:16.345183 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:16.345121    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:16.569340 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:16.569313    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:37:17.114263 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:17.114236    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-31.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:37:17.595889 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:17.595860    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-31.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:37:17.820703 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:17.820671    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pdzj2"
Oct 30 18:37:17.859199 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:17.859145    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:37:18.277179 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:18.277123    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:18.346395 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:18.346374    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:18.467338 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:18.467321    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:18.796088 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:18.796063    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:18.796088 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:18.796091    2578 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:18.824719 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:18.824687    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:32:17 +0000 UTC" deadline="2028-05-16 21:52:32.121750202 +0000 UTC"
Oct 30 18:37:18.824719 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:18.824717    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13539h15m13.297038759s"
Oct 30 18:37:18.885124 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:18.885102    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:18.934473 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:18.934453    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:18.950609 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:18.950589    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:18.990825 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:18.990794    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:19.056488 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:19.056447    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:19.376902 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:19.376882    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:19.376902 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:19.376903    2578 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:19.476232 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:19.476206    2578 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:19.688205 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:19.688158    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:19.753674 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:19.753653    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:19.859942 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:19.859923    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:20.223833 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:20.223808    2578 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:20.223833 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:20.223830    2578 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-129-31.us-west-2.compute.internal" not found
Oct 30 18:37:20.223976 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:20.223957    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:20.225928 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:20.225912    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:20.226836 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:20.226820    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:20.226908 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:20.226855    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:20.226908 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:20.226868    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:20.226908 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:20.226896    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:20.285846 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:20.285820    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:20.285846 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:20.285845    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-31.us-west-2.compute.internal\": node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:20.401371 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:20.401341    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:20.501681 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:20.501623    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:20.602166 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:20.602142    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:20.702603 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:20.702575    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:20.803290 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:20.803240    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:20.904125 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:20.904101    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:21.004956 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:21.004928    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:21.105569 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:21.105546    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:21.206075 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:21.206051    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:21.306558 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:21.306511    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:21.407369 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:21.407324    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:21.507789 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:21.507762    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:21.608284 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:21.608261    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:21.708786 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:21.708731    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:21.809268 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:21.809240    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:21.910002 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:21.909983    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:22.010925 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:22.010873    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:22.111507 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:22.111487    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:22.212042 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:22.212019    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:22.312662 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:22.312618    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:22.413233 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:22.413211    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:22.513429 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:22.513406    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:22.613879 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:22.613857    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:22.714390 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:22.714366    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:22.814870 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:22.814844    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:22.915863 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:22.915814    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:23.016589 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:23.016559    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:23.117150 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:23.117130    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:23.217614 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:23.217569    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:23.318109 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:23.318078    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:23.418851 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:23.418825    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:23.519403 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:23.519359    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:23.619869 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:23.619847    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:23.720924 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:23.720898    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:23.821446 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:23.821398    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:23.922430 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:23.922407    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:24.022745 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:24.022724    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:24.123237 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:24.123215    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:24.223990 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:24.223955    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:24.324379 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:24.324348    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:24.425146 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:24.425091    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:24.525522 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:24.525488    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:24.625977 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:24.625953    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:24.726418 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:24.726371    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:24.826953 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:24.826929    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:24.927903 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:24.927882    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:25.028670 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:25.028630    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:25.129325 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:25.129299    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:25.229837 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:25.229802    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:25.330318 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:25.330297    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:25.431122 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:25.431089    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:25.531564 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:25.531539    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:25.632121 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:25.632075    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:25.732501 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:25.732480    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:25.833005 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:25.832974    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:25.933867 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:25.933828    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:26.034587 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:26.034562    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:26.135271 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:26.135249    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:26.235740 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:26.235718    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:26.336232 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:26.336209    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:26.436454 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:26.436423    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:26.536970 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:26.536924    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:26.637372 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:26.637333    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:26.737859 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:26.737831    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:26.838793 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:26.838773    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:26.939612 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:26.939584    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:27.040254 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.040227    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:27.105385 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:27.105335    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_e1c96e16bc130f55ab259bba0b1cc606/setup/1.log"
Oct 30 18:37:27.105823 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:27.105804    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_e1c96e16bc130f55ab259bba0b1cc606/setup/0.log"
Oct 30 18:37:27.105906 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:27.105846    2578 generic.go:356] "Generic (PLEG): container finished" podID="e1c96e16bc130f55ab259bba0b1cc606" containerID="324ec2c193e25b0b6e5c1614b81a8c2aaa1409f60467d2d5a80863aabbef354f" exitCode=1
Oct 30 18:37:27.105906 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:27.105879    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"e1c96e16bc130f55ab259bba0b1cc606","Type":"ContainerDied","Data":"324ec2c193e25b0b6e5c1614b81a8c2aaa1409f60467d2d5a80863aabbef354f"}
Oct 30 18:37:27.106016 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:27.105945    2578 scope.go:122] "RemoveContainer" containerID="abdbf86c78b8cca0528108aebac6f1ba860cc40bd481cd465bd4fccb7f131697"
Oct 30 18:37:27.106064 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:27.106039    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:27.107036 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:27.107011    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:27.107154 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:27.107077    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:27.107154 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:27.107090    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:27.107513 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.107337    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:27.107513 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:27.107404    2578 scope.go:122] "RemoveContainer" containerID="abdbf86c78b8cca0528108aebac6f1ba860cc40bd481cd465bd4fccb7f131697"
Oct 30 18:37:27.116957 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.116921    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_openshift-machine-config-operator_e1c96e16bc130f55ab259bba0b1cc606_0 in pod sandbox 436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685 from index: no such id: 'abdbf86c78b8cca0528108aebac6f1ba860cc40bd481cd465bd4fccb7f131697'" containerID="abdbf86c78b8cca0528108aebac6f1ba860cc40bd481cd465bd4fccb7f131697"
Oct 30 18:37:27.117046 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.116988    2578 kuberuntime_container.go:987] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_openshift-machine-config-operator_e1c96e16bc130f55ab259bba0b1cc606_0 in pod sandbox 436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685 from index: no such id: 'abdbf86c78b8cca0528108aebac6f1ba860cc40bd481cd465bd4fccb7f131697'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_openshift-machine-config-operator(e1c96e16bc130f55ab259bba0b1cc606)\"" logger="UnhandledError"
Oct 30 18:37:27.117191 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.117171    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_openshift-machine-config-operator(e1c96e16bc130f55ab259bba0b1cc606)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" podUID="e1c96e16bc130f55ab259bba0b1cc606"
Oct 30 18:37:27.141279 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.141259    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:27.241901 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.241872    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:27.341949 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.341921    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:27.442766 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.442706    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:27.543250 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.543218    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:27.644132 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.644100    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:27.744619 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.744568    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:27.845038 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.845008    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:27.945081 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:27.945058    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:28.045749 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:28.045693    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:28.108770 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:28.108749    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_e1c96e16bc130f55ab259bba0b1cc606/setup/1.log"
Oct 30 18:37:28.145998 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:28.145982    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:28.246440 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:28.246415    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:28.347336 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:28.347320    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:28.448147 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:28.448127    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:28.548593 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:28.548573    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:28.649134 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:28.649094    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:28.749576 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:28.749555    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:28.850244 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:28.850218    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:28.951080 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:28.951020    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:28.991242 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:28.991203    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:29.052088 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:29.052065    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:29.153032 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:29.153010    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:29.253122 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:29.253075    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:29.353726 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:29.353705    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:29.454550 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:29.454515    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:29.555202 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:29.555169    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:29.656262 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:29.656241    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:29.757172 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:29.757148    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:29.857566 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:29.857543    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:29.958232 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:29.958201    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:30.058440 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:30.058419    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:30.159039 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:30.158982    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:30.259661 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:30.259638    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:30.360249 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:30.360227    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:30.461183 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:30.461146    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:30.561845 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:30.561814    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:30.662578 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:30.662521    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:30.731330 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:30.731292    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-31.us-west-2.compute.internal\": node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:30.855605 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:30.855577    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:30.956617 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:30.956595    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:31.057600 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:31.057557    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:31.158396 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:31.158372    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:31.259292 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:31.259273    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:31.359361 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:31.359339    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:31.459431 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:31.459405    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:31.559945 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:31.559926    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:31.660430 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:31.660385    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:31.761290 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:31.761269    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:31.861937 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:31.861915    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:31.868634 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:31.868616    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:31.962490 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:31.962447    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:32.062779 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:32.062760    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:32.163219 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:32.163199    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:32.263692 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:32.263648    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:32.364367 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:32.364338    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:32.465168 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:32.465148    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:32.565999 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:32.565963    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:32.666891 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:32.666861    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:32.767557 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:32.767523    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:32.868517 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:32.868491    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:32.969513 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:32.969488    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:33.070448 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:33.070430    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:33.171045 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:33.170984    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:33.271652 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:33.271624    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:33.371735 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:33.371713    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:33.472334 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:33.472292    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:33.573336 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:33.573311    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:33.674212 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:33.674185    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:33.774680 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:33.774636    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:33.875631 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:33.875609    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:33.976449 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:33.976427    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:34.076892 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:34.076875    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:34.177392 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:34.177367    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:34.277785 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:34.277758    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:34.378306 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:34.378265    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:34.478666 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:34.478641    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:34.579043 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:34.579024    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:34.679464 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:34.679418    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:34.691586 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:34.691568    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:37:34.700148 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:34.700129    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:37:34.700245 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:34.700163    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:37:34.700245 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:34.700175    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:37:34.780542 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:34.780510    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:34.881121 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:34.881085    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:34.981981 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:34.981937    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:35.082682 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:35.082662    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:35.183038 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:35.183006    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:35.283649 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:35.283612    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:35.384518 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:35.384501    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:35.485344 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:35.485323    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:37:35.527991 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:35.527972    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:37:35.624098 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:35.624075    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:35.698576 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:35.698557    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:37:35.698684 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:35.698670    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:37:35.792337 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:35.792317    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:37:35.921076 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:35.921030    2578 apiserver.go:52] "Watching apiserver"
Oct 30 18:37:35.944497 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:35.944477    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:37:36.080646 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.080625    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:37:36.081020 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.080998    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-7k5m2","openshift-dns/node-resolver-tnkkl","openshift-multus/multus-26flh","openshift-multus/multus-additional-cni-plugins-nfgck","kube-system/konnectivity-agent-25n6m","kube-system/global-pull-secret-syncer-mqdgz","openshift-image-registry/node-ca-q4mnk","openshift-multus/network-metrics-daemon-w8jdh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7","openshift-network-diagnostics/network-check-target-v7pb2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal","openshift-monitoring/node-exporter-bjkm5","openshift-network-operator/iptables-alerter-hhdgg","openshift-cluster-node-tuning-operator/tuned-ckgwd","kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"]
Oct 30 18:37:36.084967 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.084951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:37:36.087134 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.087109    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:36.087244 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.087220    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:36.089283 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.089259    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.091563 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.091544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.093770 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.093747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:36.096039 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.096018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:36.096117 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.096090    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:36.098352 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.098332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:37:36.100593 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.100574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.103137 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.103121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.106482 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.106452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:36.106605 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.106517    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:36.109287 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.109267    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.111541 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.111511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:37:36.113940 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.113922    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.114940 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.114914    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_openshift-machine-config-operator(e1c96e16bc130f55ab259bba0b1cc606)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" podUID="e1c96e16bc130f55ab259bba0b1cc606"
Oct 30 18:37:36.125793 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.125775    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:37:36.134167 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-wtmp\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.134241 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-bin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.134241 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-daemon-config\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.134327 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-config\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.134327 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-script-lib\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.134327 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.134327 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-sys-fs\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.134468 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-netns\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.134468 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:36.134468 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-slash\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.134468 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-env-overrides\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.134468 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-host\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:37:36.134468 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.134468 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-tmp\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.134694 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.134694 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-cnibin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.134694 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-conf-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.134694 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-serviceca\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:37:36.134694 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-run\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.134694 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk6ls\" (UniqueName: \"kubernetes.io/projected/25811b4f-a869-49f0-ac40-b0f37274e97d-kube-api-access-zk6ls\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.134694 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-sys\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.134694 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-metrics-client-ca\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.134694 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-kubelet\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.134694 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysconfig\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lggv5\" (UniqueName: \"kubernetes.io/projected/2bd81706-d26f-420f-a97d-270ebbc96057-kube-api-access-lggv5\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jkqx\" (UniqueName: \"kubernetes.io/projected/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-kube-api-access-7jkqx\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-etc-kubernetes\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-socket-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-modprobe-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-sys\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-textfile\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-cni-binary-copy\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xwf7\" (UniqueName: \"kubernetes.io/projected/6b567074-99fd-4062-9fbe-a356eb37fe26-kube-api-access-6xwf7\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.134987 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-etc-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.134989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-cnibin\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-device-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7zhh\" (UniqueName: \"kubernetes.io/projected/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-kube-api-access-j7zhh\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47d77dc5-6269-4949-b4fa-60c617073f7c-agent-certs\") pod \"konnectivity-agent-25n6m\" (UID: \"47d77dc5-6269-4949-b4fa-60c617073f7c\") " pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-tmp-dir\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-kubernetes\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-netd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-system-cni-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf42g\" (UniqueName: \"kubernetes.io/projected/cb8aaa1a-7cea-437a-a436-09a8af560781-kube-api-access-cf42g\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-hosts-file\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-lib-modules\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgxv8\" (UniqueName: \"kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8\") pod \"network-check-target-v7pb2\" (UID: \"de6f539e-a438-42bb-98e6-d83b5926143a\") " pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-dbus\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:36.135390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-ovn\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-node-log\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshdb\" (UniqueName: \"kubernetes.io/projected/8bc74796-475a-4fe8-9113-a6dacb274acd-kube-api-access-lshdb\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-systemd\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-tuned\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-hostroot\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-systemd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-iptables-alerter-script\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-tls\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-kubelet\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-host-slash\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-var-lib-kubelet\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-multus\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-log-socket\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47d77dc5-6269-4949-b4fa-60c617073f7c-konnectivity-ca\") pod \"konnectivity-agent-25n6m\" (UID: \"47d77dc5-6269-4949-b4fa-60c617073f7c\") " pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:36.136071 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfpk7\" (UniqueName: \"kubernetes.io/projected/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-kube-api-access-dfpk7\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-conf\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-accelerators-collector-config\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.135978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-socket-dir-parent\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-k8s-cni-cncf-io\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-multus-certs\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-root\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-registration-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffxdt\" (UniqueName: \"kubernetes.io/projected/62eff799-47b9-474a-966f-2689ab4f1c17-kube-api-access-ffxdt\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-system-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-kubelet-config\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-bin\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-etc-selinux\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.136685 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-host\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.137212 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-systemd-units\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.137212 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-netns\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.137212 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-var-lib-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.137212 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bc74796-475a-4fe8-9113-a6dacb274acd-ovn-node-metrics-cert\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.137212 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74dqb\" (UniqueName: \"kubernetes.io/projected/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-kube-api-access-74dqb\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:37:36.137212 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.136395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.140693 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.140675    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:37:36.140790 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.140728    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:37:36.140790 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.140773    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:37:36.141012 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.140997    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sslwz\""
Oct 30 18:37:36.236102 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.236050    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:37:36.236102 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.236082    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hs2sz"
Oct 30 18:37:36.237145 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jkqx\" (UniqueName: \"kubernetes.io/projected/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-kube-api-access-7jkqx\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:36.237199 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-etc-kubernetes\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.237238 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-socket-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.237238 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-modprobe-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.237293 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-sys\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.237293 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-textfile\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.237293 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-cni-binary-copy\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.237392 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwf7\" (UniqueName: \"kubernetes.io/projected/6b567074-99fd-4062-9fbe-a356eb37fe26-kube-api-access-6xwf7\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.237392 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-etc-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.237392 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.237392 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-cnibin\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-device-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237419    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-etc-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237419    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-modprobe-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237460    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-device-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237463    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237461    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-etc-kubernetes\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-sys\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237511    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-cnibin\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7zhh\" (UniqueName: \"kubernetes.io/projected/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-kube-api-access-j7zhh\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237575    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-socket-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47d77dc5-6269-4949-b4fa-60c617073f7c-agent-certs\") pod \"konnectivity-agent-25n6m\" (UID: \"47d77dc5-6269-4949-b4fa-60c617073f7c\") " pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:36.237601 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237593    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-textfile\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-tmp-dir\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-kubernetes\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-netd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-system-cni-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237686    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-kubernetes\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf42g\" (UniqueName: \"kubernetes.io/projected/cb8aaa1a-7cea-437a-a436-09a8af560781-kube-api-access-cf42g\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237739    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-system-cni-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-hosts-file\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237737    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-netd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-lib-modules\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237798    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-hosts-file\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxv8\" (UniqueName: \"kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8\") pod \"network-check-target-v7pb2\" (UID: \"de6f539e-a438-42bb-98e6-d83b5926143a\") " pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-dbus\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-ovn\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237875    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-tmp-dir\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:37:36.238115 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-node-log\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237922    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-node-log\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-lib-modules\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshdb\" (UniqueName: \"kubernetes.io/projected/8bc74796-475a-4fe8-9113-a6dacb274acd-kube-api-access-lshdb\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237930    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-ovn\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-systemd\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-tuned\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237989    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-systemd\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.237986    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-dbus\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-hostroot\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-systemd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-iptables-alerter-script\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238077    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-hostroot\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238090    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-systemd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-tls\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-kubelet\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238171    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-kubelet\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.238765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-host-slash\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238199    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-var-lib-kubelet\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238240    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-host-slash\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-multus\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238248    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-var-lib-kubelet\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238271    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-multus\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-log-socket\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238287    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238296    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-log-socket\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47d77dc5-6269-4949-b4fa-60c617073f7c-konnectivity-ca\") pod \"konnectivity-agent-25n6m\" (UID: \"47d77dc5-6269-4949-b4fa-60c617073f7c\") " pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfpk7\" (UniqueName: \"kubernetes.io/projected/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-kube-api-access-dfpk7\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-conf\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-accelerators-collector-config\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-socket-dir-parent\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238457    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-conf\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-k8s-cni-cncf-io\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-multus-certs\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.239396 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238499    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-socket-dir-parent\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238524    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-k8s-cni-cncf-io\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-root\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238552    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-multus-certs\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238593    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-root\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-registration-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffxdt\" (UniqueName: \"kubernetes.io/projected/62eff799-47b9-474a-966f-2689ab4f1c17-kube-api-access-ffxdt\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238650    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-registration-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-system-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-kubelet-config\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.238718    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238735    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-bin\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-bin\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238773    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-system-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238778    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-kubelet-config\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.238801    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs podName:e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:36.738784804 +0000 UTC m=+38.238788423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs") pod "network-metrics-daemon-w8jdh" (UID: "e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-etc-selinux\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-host\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-systemd-units\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-netns\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238880    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-host\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238883    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-etc-selinux\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-var-lib-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238907    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-systemd-units\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-netns\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238930    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-var-lib-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bc74796-475a-4fe8-9113-a6dacb274acd-ovn-node-metrics-cert\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.238969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74dqb\" (UniqueName: \"kubernetes.io/projected/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-kube-api-access-74dqb\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239095    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-wtmp\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.240705 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-bin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-daemon-config\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-config\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-script-lib\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239190    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-bin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-sys-fs\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-netns\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239284    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-wtmp\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239286    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-slash\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239313    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-netns\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239324    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-sys-fs\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-env-overrides\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239361    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-slash\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.239371    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:36.241478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-host\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.239409    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret podName:b81698fa-f5d9-48b0-85fa-29e71a85c441 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:36.739394914 +0000 UTC m=+38.239398506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret") pod "global-pull-secret-syncer-mqdgz" (UID: "b81698fa-f5d9-48b0-85fa-29e71a85c441") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239423    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-host\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-tmp\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-cnibin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-conf-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-serviceca\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239646    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-run\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239679    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-cnibin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239686    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6ls\" (UniqueName: \"kubernetes.io/projected/25811b4f-a869-49f0-ac40-b0f37274e97d-kube-api-access-zk6ls\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239700    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-run\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239700    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-conf-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-sys\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.242266 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-metrics-client-ca\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-kubelet\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239782    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-sys\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239795    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239813    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-kubelet\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysconfig\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239845    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lggv5\" (UniqueName: \"kubernetes.io/projected/2bd81706-d26f-420f-a97d-270ebbc96057-kube-api-access-lggv5\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.239876    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysconfig\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.240642    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.240821    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-serviceca\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.240934    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.240999    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241137    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241194    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241247    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241264    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241340    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241349    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241354    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241270    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241349    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241496    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241740    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241794    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241902    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:37:36.242806 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241968    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:37:36.243478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.241991    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:37:36.243478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.242050    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:37:36.243478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.242066    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:37:36.243478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.242093    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:37:36.243478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.242132    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:37:36.243478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.242200    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:37:36.243478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.242236    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:37:36.243478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.242250    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:37:36.243478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.242616    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-tmp\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.243478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.242725    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-tuned\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.248376 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.248358    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-cni-binary-copy\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.250595 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.250150    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-iptables-alerter-script\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:37:36.250595 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.250162    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47d77dc5-6269-4949-b4fa-60c617073f7c-konnectivity-ca\") pod \"konnectivity-agent-25n6m\" (UID: \"47d77dc5-6269-4949-b4fa-60c617073f7c\") " pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:36.250595 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.250436    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-daemon-config\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.250595 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.250446    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-accelerators-collector-config\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.250595 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.250470    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.250595 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.250492    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-config\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.250595 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.250567    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.250825 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.250742    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.250825 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.250750    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-metrics-client-ca\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.250881 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.250870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-env-overrides\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.250957 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.250935    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-script-lib\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.257410 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.257390    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:37:36.258794 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.258774    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:37:36.258875 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.258805    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:37:36.260780 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.260754    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.261580 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.261559    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bc74796-475a-4fe8-9113-a6dacb274acd-ovn-node-metrics-cert\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.287592 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.287577    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thsk7\""
Oct 30 18:37:36.287725 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.287711    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-djndp\""
Oct 30 18:37:36.292332 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.292320    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vmn22\""
Oct 30 18:37:36.292374 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.292330    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k6xhn\""
Oct 30 18:37:36.292628 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.292613    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:37:36.292684 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.292662    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7txc\""
Oct 30 18:37:36.292684 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.292670    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w6mkd\""
Oct 30 18:37:36.292684 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.292676    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6qrp7\""
Oct 30 18:37:36.292808 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.292680    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:37:36.292808 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.292671    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p85r8\""
Oct 30 18:37:36.292808 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.292717    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h66ck\""
Oct 30 18:37:36.300660 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.300635    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-tls\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.300748 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.300734    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47d77dc5-6269-4949-b4fa-60c617073f7c-agent-certs\") pod \"konnectivity-agent-25n6m\" (UID: \"47d77dc5-6269-4949-b4fa-60c617073f7c\") " pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:36.315937 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.315917    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:36.315937 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.315936    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:36.316035 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.315945    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sgxv8 for pod openshift-network-diagnostics/network-check-target-v7pb2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:36.316035 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.315986    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8 podName:de6f539e-a438-42bb-98e6-d83b5926143a nodeName:}" failed. No retries permitted until 2026-10-30 18:37:36.815974599 +0000 UTC m=+38.315978235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sgxv8" (UniqueName: "kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8") pod "network-check-target-v7pb2" (UID: "de6f539e-a438-42bb-98e6-d83b5926143a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:36.325176 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.325152    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfpk7\" (UniqueName: \"kubernetes.io/projected/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-kube-api-access-dfpk7\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:37:36.325176 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.325168    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6ls\" (UniqueName: \"kubernetes.io/projected/25811b4f-a869-49f0-ac40-b0f37274e97d-kube-api-access-zk6ls\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.325323 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.325173    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf42g\" (UniqueName: \"kubernetes.io/projected/cb8aaa1a-7cea-437a-a436-09a8af560781-kube-api-access-cf42g\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.325323 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.325152    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7zhh\" (UniqueName: \"kubernetes.io/projected/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-kube-api-access-j7zhh\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:37:36.325323 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.325251    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lggv5\" (UniqueName: \"kubernetes.io/projected/2bd81706-d26f-420f-a97d-270ebbc96057-kube-api-access-lggv5\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.325323 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.325299    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74dqb\" (UniqueName: \"kubernetes.io/projected/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-kube-api-access-74dqb\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:37:36.325448 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.325440    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwf7\" (UniqueName: \"kubernetes.io/projected/6b567074-99fd-4062-9fbe-a356eb37fe26-kube-api-access-6xwf7\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.325480 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.325470    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshdb\" (UniqueName: \"kubernetes.io/projected/8bc74796-475a-4fe8-9113-a6dacb274acd-kube-api-access-lshdb\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.325953 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.325934    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffxdt\" (UniqueName: \"kubernetes.io/projected/62eff799-47b9-474a-966f-2689ab4f1c17-kube-api-access-ffxdt\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.326832 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.326815    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jkqx\" (UniqueName: \"kubernetes.io/projected/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-kube-api-access-7jkqx\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:36.397117 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.397098    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:37:36.403139 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.403121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-26flh"
Oct 30 18:37:36.406723 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:37:36.406703    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23b1354_07f4_4b59_a6a1_a1c96c8c6b27.slice/crio-ede44013528faa80e66e24b7e962fb69a7f308c016ac52540affa302fabfdb69 WatchSource:0}: Error finding container ede44013528faa80e66e24b7e962fb69a7f308c016ac52540affa302fabfdb69: Status 404 returned error can't find the container with id ede44013528faa80e66e24b7e962fb69a7f308c016ac52540affa302fabfdb69
Oct 30 18:37:36.411765 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:37:36.411745    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b567074_99fd_4062_9fbe_a356eb37fe26.slice/crio-8ce11b8efa0c0a4d42e9503ad491db65e2ea21a568c781e349cc65819153bd2f WatchSource:0}: Error finding container 8ce11b8efa0c0a4d42e9503ad491db65e2ea21a568c781e349cc65819153bd2f: Status 404 returned error can't find the container with id 8ce11b8efa0c0a4d42e9503ad491db65e2ea21a568c781e349cc65819153bd2f
Oct 30 18:37:36.422776 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.422758    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:36.424884 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.424868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:37:36.426144 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.426127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:37:36.428282 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.428196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:37:36.431505 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:37:36.431487    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d77dc5_6269_4949_b4fa_60c617073f7c.slice/crio-398b4da201df26917b4d3f8cd1286a2c955b47eb1e754b955c106996dba51846 WatchSource:0}: Error finding container 398b4da201df26917b4d3f8cd1286a2c955b47eb1e754b955c106996dba51846: Status 404 returned error can't find the container with id 398b4da201df26917b4d3f8cd1286a2c955b47eb1e754b955c106996dba51846
Oct 30 18:37:36.434478 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:37:36.434461    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c2b89c_c9e8_4bc7_a733_e9ba76cde59c.slice/crio-866ea6c26969fd3288b62075097aa4e24eead5dcdb9c1c8194822a8fe46b646f WatchSource:0}: Error finding container 866ea6c26969fd3288b62075097aa4e24eead5dcdb9c1c8194822a8fe46b646f: Status 404 returned error can't find the container with id 866ea6c26969fd3288b62075097aa4e24eead5dcdb9c1c8194822a8fe46b646f
Oct 30 18:37:36.440063 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:37:36.440046    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25811b4f_a869_49f0_ac40_b0f37274e97d.slice/crio-d77b8dc24e7dfbf77ec685a81da26a158cbe01837f2eb47d5468b9084015d434 WatchSource:0}: Error finding container d77b8dc24e7dfbf77ec685a81da26a158cbe01837f2eb47d5468b9084015d434: Status 404 returned error can't find the container with id d77b8dc24e7dfbf77ec685a81da26a158cbe01837f2eb47d5468b9084015d434
Oct 30 18:37:36.441313 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:37:36.441292    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8aaa1a_7cea_437a_a436_09a8af560781.slice/crio-71011d241d3eef21a5ef163b81b0569dd94f02594b79ea8c7027bc267cf11c16 WatchSource:0}: Error finding container 71011d241d3eef21a5ef163b81b0569dd94f02594b79ea8c7027bc267cf11c16: Status 404 returned error can't find the container with id 71011d241d3eef21a5ef163b81b0569dd94f02594b79ea8c7027bc267cf11c16
Oct 30 18:37:36.450668 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.450646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:37:36.451764 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.451743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:37:36.456323 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.456301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:36.460012 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.459993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:37:36.462084 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:37:36.462062    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3bc506_2acc_4de7_8d28_738da0eaa6ee.slice/crio-642a234072bbe9e5423e488fd3d5b99fcd9d9b8aebdd0385cadebbde7b2a3b77 WatchSource:0}: Error finding container 642a234072bbe9e5423e488fd3d5b99fcd9d9b8aebdd0385cadebbde7b2a3b77: Status 404 returned error can't find the container with id 642a234072bbe9e5423e488fd3d5b99fcd9d9b8aebdd0385cadebbde7b2a3b77
Oct 30 18:37:36.463377 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:37:36.463309    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62eff799_47b9_474a_966f_2689ab4f1c17.slice/crio-f67508545cfba1c53d9bee04cb2bbd24cd4db60605f77f023f2f210e040eb33d WatchSource:0}: Error finding container f67508545cfba1c53d9bee04cb2bbd24cd4db60605f77f023f2f210e040eb33d: Status 404 returned error can't find the container with id f67508545cfba1c53d9bee04cb2bbd24cd4db60605f77f023f2f210e040eb33d
Oct 30 18:37:36.465248 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:37:36.465216    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc74796_475a_4fe8_9113_a6dacb274acd.slice/crio-3158b9688a04e6027f1c4e6594c2a025483d8bae4bb9dcac1c86b6982e7084f7 WatchSource:0}: Error finding container 3158b9688a04e6027f1c4e6594c2a025483d8bae4bb9dcac1c86b6982e7084f7: Status 404 returned error can't find the container with id 3158b9688a04e6027f1c4e6594c2a025483d8bae4bb9dcac1c86b6982e7084f7
Oct 30 18:37:36.468740 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:37:36.468721    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd81706_d26f_420f_a97d_270ebbc96057.slice/crio-bab21f8676dafd9bbdd1d1c7f2ede6953d0b9dbe26ca1b7275fdfeed42856b8f WatchSource:0}: Error finding container bab21f8676dafd9bbdd1d1c7f2ede6953d0b9dbe26ca1b7275fdfeed42856b8f: Status 404 returned error can't find the container with id bab21f8676dafd9bbdd1d1c7f2ede6953d0b9dbe26ca1b7275fdfeed42856b8f
Oct 30 18:37:36.473110 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.473078    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal" podStartSLOduration=1.473068576 podStartE2EDuration="1.473068576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:36.355196638 +0000 UTC m=+37.855200254" watchObservedRunningTime="2026-10-30 18:37:36.473068576 +0000 UTC m=+37.973072191"
Oct 30 18:37:36.743410 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.743387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:36.743725 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.743419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:36.743725 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.743496    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:36.743725 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.743513    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:36.743725 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.743547    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret podName:b81698fa-f5d9-48b0-85fa-29e71a85c441 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:37.743522969 +0000 UTC m=+39.243526561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret") pod "global-pull-secret-syncer-mqdgz" (UID: "b81698fa-f5d9-48b0-85fa-29e71a85c441") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:36.743725 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.743575    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs podName:e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:37.743562958 +0000 UTC m=+39.243566551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs") pod "network-metrics-daemon-w8jdh" (UID: "e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:36.843786 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:36.843769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxv8\" (UniqueName: \"kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8\") pod \"network-check-target-v7pb2\" (UID: \"de6f539e-a438-42bb-98e6-d83b5926143a\") " pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:36.843885 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.843858    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:36.843885 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.843869    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:36.843885 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.843877    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sgxv8 for pod openshift-network-diagnostics/network-check-target-v7pb2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:36.843984 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:36.843912    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8 podName:de6f539e-a438-42bb-98e6-d83b5926143a nodeName:}" failed. No retries permitted until 2026-10-30 18:37:37.84390193 +0000 UTC m=+39.343905537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sgxv8" (UniqueName: "kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8") pod "network-check-target-v7pb2" (UID: "de6f539e-a438-42bb-98e6-d83b5926143a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:37.122429 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.122230    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q4mnk" event={"ID":"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27","Type":"ContainerStarted","Data":"ede44013528faa80e66e24b7e962fb69a7f308c016ac52540affa302fabfdb69"}
Oct 30 18:37:37.123515 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.123482    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bjkm5" event={"ID":"2bd81706-d26f-420f-a97d-270ebbc96057","Type":"ContainerStarted","Data":"bab21f8676dafd9bbdd1d1c7f2ede6953d0b9dbe26ca1b7275fdfeed42856b8f"}
Oct 30 18:37:37.124671 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.124644    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2" event={"ID":"8bc74796-475a-4fe8-9113-a6dacb274acd","Type":"ContainerStarted","Data":"3158b9688a04e6027f1c4e6594c2a025483d8bae4bb9dcac1c86b6982e7084f7"}
Oct 30 18:37:37.125822 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.125797    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7" event={"ID":"62eff799-47b9-474a-966f-2689ab4f1c17","Type":"ContainerStarted","Data":"f67508545cfba1c53d9bee04cb2bbd24cd4db60605f77f023f2f210e040eb33d"}
Oct 30 18:37:37.126811 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.126791    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfgck" event={"ID":"cb8aaa1a-7cea-437a-a436-09a8af560781","Type":"ContainerStarted","Data":"71011d241d3eef21a5ef163b81b0569dd94f02594b79ea8c7027bc267cf11c16"}
Oct 30 18:37:37.127786 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.127764    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tnkkl" event={"ID":"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c","Type":"ContainerStarted","Data":"866ea6c26969fd3288b62075097aa4e24eead5dcdb9c1c8194822a8fe46b646f"}
Oct 30 18:37:37.128750 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.128717    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-25n6m" event={"ID":"47d77dc5-6269-4949-b4fa-60c617073f7c","Type":"ContainerStarted","Data":"398b4da201df26917b4d3f8cd1286a2c955b47eb1e754b955c106996dba51846"}
Oct 30 18:37:37.129804 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.129785    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-26flh" event={"ID":"6b567074-99fd-4062-9fbe-a356eb37fe26","Type":"ContainerStarted","Data":"8ce11b8efa0c0a4d42e9503ad491db65e2ea21a568c781e349cc65819153bd2f"}
Oct 30 18:37:37.130738 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.130718    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hhdgg" event={"ID":"ed3bc506-2acc-4de7-8d28-738da0eaa6ee","Type":"ContainerStarted","Data":"642a234072bbe9e5423e488fd3d5b99fcd9d9b8aebdd0385cadebbde7b2a3b77"}
Oct 30 18:37:37.131624 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.131605    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ckgwd" event={"ID":"25811b4f-a869-49f0-ac40-b0f37274e97d","Type":"ContainerStarted","Data":"d77b8dc24e7dfbf77ec685a81da26a158cbe01837f2eb47d5468b9084015d434"}
Oct 30 18:37:37.239761 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.239738    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:36 +0000 UTC" deadline="2028-07-03 04:27:41.672780483 +0000 UTC"
Oct 30 18:37:37.239761 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.239760    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14673h50m4.433023302s"
Oct 30 18:37:37.752043 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.751569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:37.752043 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.751609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:37.752043 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:37.751715    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:37.752043 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:37.751720    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:37.752043 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:37.751756    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret podName:b81698fa-f5d9-48b0-85fa-29e71a85c441 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:39.751744297 +0000 UTC m=+41.251747902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret") pod "global-pull-secret-syncer-mqdgz" (UID: "b81698fa-f5d9-48b0-85fa-29e71a85c441") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:37.752043 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:37.751779    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs podName:e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:39.751761449 +0000 UTC m=+41.251765084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs") pod "network-metrics-daemon-w8jdh" (UID: "e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:37.852933 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:37.852897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxv8\" (UniqueName: \"kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8\") pod \"network-check-target-v7pb2\" (UID: \"de6f539e-a438-42bb-98e6-d83b5926143a\") " pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:37.853147 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:37.853096    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:37.853147 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:37.853119    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:37.853147 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:37.853131    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sgxv8 for pod openshift-network-diagnostics/network-check-target-v7pb2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:37.853314 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:37.853175    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8 podName:de6f539e-a438-42bb-98e6-d83b5926143a nodeName:}" failed. No retries permitted until 2026-10-30 18:37:39.853163629 +0000 UTC m=+41.353167236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sgxv8" (UniqueName: "kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8") pod "network-check-target-v7pb2" (UID: "de6f539e-a438-42bb-98e6-d83b5926143a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:38.056358 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:38.056284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:38.056513 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:38.056402    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:38.056668 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:38.056647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:38.056789 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:38.056766    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:38.056789 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:38.056784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:38.056884 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:38.056870    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:38.159629 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:38.159602    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_e1c96e16bc130f55ab259bba0b1cc606/setup/1.log"
Oct 30 18:37:38.159798 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:38.159645    2578 generic.go:356] "Generic (PLEG): container finished" podID="e1c96e16bc130f55ab259bba0b1cc606" containerID="167cbe67b9c7afdfff5902fce31928cc091cb6900361b21f8442ba10eb4cd531" exitCode=0
Oct 30 18:37:38.159798 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:38.159677    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"e1c96e16bc130f55ab259bba0b1cc606","Type":"ContainerDied","Data":"167cbe67b9c7afdfff5902fce31928cc091cb6900361b21f8442ba10eb4cd531"}
Oct 30 18:37:38.159798 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:38.159719    2578 scope.go:122] "RemoveContainer" containerID="324ec2c193e25b0b6e5c1614b81a8c2aaa1409f60467d2d5a80863aabbef354f"
Oct 30 18:37:38.160258 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:38.160164    2578 scope.go:122] "RemoveContainer" containerID="324ec2c193e25b0b6e5c1614b81a8c2aaa1409f60467d2d5a80863aabbef354f"
Oct 30 18:37:38.229610 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:38.229329    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_openshift-machine-config-operator_e1c96e16bc130f55ab259bba0b1cc606_1 in pod sandbox 436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685 from index: no such id: '324ec2c193e25b0b6e5c1614b81a8c2aaa1409f60467d2d5a80863aabbef354f'" containerID="324ec2c193e25b0b6e5c1614b81a8c2aaa1409f60467d2d5a80863aabbef354f"
Oct 30 18:37:38.229610 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:38.229375    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324ec2c193e25b0b6e5c1614b81a8c2aaa1409f60467d2d5a80863aabbef354f"} err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal_openshift-machine-config-operator_e1c96e16bc130f55ab259bba0b1cc606_1 in pod sandbox 436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685 from index: no such id: '324ec2c193e25b0b6e5c1614b81a8c2aaa1409f60467d2d5a80863aabbef354f'"
Oct 30 18:37:38.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:38.240076    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:36 +0000 UTC" deadline="2028-05-02 19:14:34.759806567 +0000 UTC"
Oct 30 18:37:38.240125 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:38.240122    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13200h36m56.519687825s"
Oct 30 18:37:39.177840 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:39.177802    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"e1c96e16bc130f55ab259bba0b1cc606","Type":"ContainerStarted","Data":"8261fc3a4272581b5efef3c863d9f6fa4c9b1bc9b4563f75c8c1fbe08f5634c2"}
Oct 30 18:37:39.774731 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:39.774307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:39.774731 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:39.774364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:39.774731 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:39.774538    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:39.774731 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:39.774595    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret podName:b81698fa-f5d9-48b0-85fa-29e71a85c441 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:43.774577171 +0000 UTC m=+45.274580794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret") pod "global-pull-secret-syncer-mqdgz" (UID: "b81698fa-f5d9-48b0-85fa-29e71a85c441") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:39.774731 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:39.774659    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:39.774731 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:39.774691    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs podName:e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:43.774680987 +0000 UTC m=+45.274684594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs") pod "network-metrics-daemon-w8jdh" (UID: "e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:39.875049 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:39.874739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxv8\" (UniqueName: \"kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8\") pod \"network-check-target-v7pb2\" (UID: \"de6f539e-a438-42bb-98e6-d83b5926143a\") " pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:39.875049 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:39.874926    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:39.875049 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:39.874948    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:39.875049 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:39.874961    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sgxv8 for pod openshift-network-diagnostics/network-check-target-v7pb2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:39.875049 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:39.875014    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8 podName:de6f539e-a438-42bb-98e6-d83b5926143a nodeName:}" failed. No retries permitted until 2026-10-30 18:37:43.874997482 +0000 UTC m=+45.375001081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sgxv8" (UniqueName: "kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8") pod "network-check-target-v7pb2" (UID: "de6f539e-a438-42bb-98e6-d83b5926143a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:40.056903 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:40.056613    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:40.056903 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:40.056635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:40.056903 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:40.056732    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:40.056903 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:40.056796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:40.056903 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:40.056874    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:40.057258 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:40.056971    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:40.191775 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:40.191736    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="8c692a768959ee16fed8810d4f88d32c87bbfe2edcc2b2112fbb0b1db6c5222f" exitCode=0
Oct 30 18:37:40.192491 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:40.191837    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfgck" event={"ID":"cb8aaa1a-7cea-437a-a436-09a8af560781","Type":"ContainerDied","Data":"8c692a768959ee16fed8810d4f88d32c87bbfe2edcc2b2112fbb0b1db6c5222f"}
Oct 30 18:37:40.196477 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:40.196451    2578 generic.go:356] "Generic (PLEG): container finished" podID="2bd81706-d26f-420f-a97d-270ebbc96057" containerID="945aa1e8973a743ec68bcc66c0ee4dc6b4284a60f05e0aef86fa3be052ea4e77" exitCode=0
Oct 30 18:37:40.196583 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:40.196492    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bjkm5" event={"ID":"2bd81706-d26f-420f-a97d-270ebbc96057","Type":"ContainerDied","Data":"945aa1e8973a743ec68bcc66c0ee4dc6b4284a60f05e0aef86fa3be052ea4e77"}
Oct 30 18:37:40.337329 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:40.337278    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" podStartSLOduration=5.337263296 podStartE2EDuration="5.337263296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:39.299745138 +0000 UTC m=+40.799748755" watchObservedRunningTime="2026-10-30 18:37:40.337263296 +0000 UTC m=+41.837266913"
Oct 30 18:37:41.208974 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:41.208902    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bjkm5" event={"ID":"2bd81706-d26f-420f-a97d-270ebbc96057","Type":"ContainerStarted","Data":"1ab08a8b9f404fb2e89df9b98fb1ec2908bb2436d572def11f78d8768eb917c3"}
Oct 30 18:37:41.209590 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:41.208980    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bjkm5" event={"ID":"2bd81706-d26f-420f-a97d-270ebbc96057","Type":"ContainerStarted","Data":"4eef611f76ce03573679fa8558dc5b3718430763232ea2a2eecf1b363ebd3932"}
Oct 30 18:37:41.216839 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:41.216800    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-25n6m" event={"ID":"47d77dc5-6269-4949-b4fa-60c617073f7c","Type":"ContainerStarted","Data":"d5a81c206a8c902736d23863652de507eb9cdfd5d5c5adf7ed76264d4a1470a4"}
Oct 30 18:37:41.424018 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:41.423865    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:41.424522 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:41.424477    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:41.459503 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:41.458460    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bjkm5" podStartSLOduration=8.516318002 podStartE2EDuration="11.458444791s" totalImagesPullingTime="2.856416656s" totalInitContainerRuntime="85.710133ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:41.336679221 +0000 UTC m=+42.836682837" watchObservedRunningTime="2026-10-30 18:37:41.458444791 +0000 UTC m=+42.958448406"
Oct 30 18:37:41.581236 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:41.580869    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-25n6m" podStartSLOduration=17.027113517 podStartE2EDuration="21.580849388s" totalImagesPullingTime="4.553735871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:41.459364991 +0000 UTC m=+42.959368607" watchObservedRunningTime="2026-10-30 18:37:41.580849388 +0000 UTC m=+43.080853008"
Oct 30 18:37:41.648621 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:41.648507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:42.056766 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:42.056007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:42.056766 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:42.056052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:42.056766 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:42.056161    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:42.056766 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:42.056570    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:42.056766 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:42.056643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:42.056766 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:42.056732    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:42.222306 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:42.222091    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:42.223671 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:42.223471    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:37:42.792401 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:42.792166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:43.225561 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:43.225504    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q4mnk" event={"ID":"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27","Type":"ContainerStarted","Data":"6719d6d2fed78c6f09fc73197660b6d97168dbe37875ec1f0bde19d633870c70"}
Oct 30 18:37:43.229606 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:43.229517    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7" event={"ID":"62eff799-47b9-474a-966f-2689ab4f1c17","Type":"ContainerStarted","Data":"01ca784f0092181e670dc7dc295486f5ea42d54e5f60a9eb68e170d94a366092"}
Oct 30 18:37:43.557445 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:43.557283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:43.812700 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:43.811892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:43.812700 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:43.811954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:43.812700 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:43.812113    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:43.812700 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:43.812175    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret podName:b81698fa-f5d9-48b0-85fa-29e71a85c441 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:51.812153317 +0000 UTC m=+53.312156931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret") pod "global-pull-secret-syncer-mqdgz" (UID: "b81698fa-f5d9-48b0-85fa-29e71a85c441") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:43.812700 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:43.812586    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:43.812700 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:43.812637    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs podName:e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:51.812621947 +0000 UTC m=+53.312625548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs") pod "network-metrics-daemon-w8jdh" (UID: "e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:43.913311 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:43.913003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxv8\" (UniqueName: \"kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8\") pod \"network-check-target-v7pb2\" (UID: \"de6f539e-a438-42bb-98e6-d83b5926143a\") " pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:43.913311 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:43.913194    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:43.913311 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:43.913212    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:43.913311 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:43.913224    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sgxv8 for pod openshift-network-diagnostics/network-check-target-v7pb2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:43.913311 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:43.913275    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8 podName:de6f539e-a438-42bb-98e6-d83b5926143a nodeName:}" failed. No retries permitted until 2026-10-30 18:37:51.913258588 +0000 UTC m=+53.413262181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sgxv8" (UniqueName: "kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8") pod "network-check-target-v7pb2" (UID: "de6f539e-a438-42bb-98e6-d83b5926143a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:44.057223 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:44.056418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:44.057223 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:44.056566    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:44.057223 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:44.056944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:44.057223 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:44.057049    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:44.057223 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:44.057101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:44.057223 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:44.057178    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:44.846871 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:44.846843    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:37:45.018883 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:45.018769    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:37:44.84687045Z","UUID":"de1b557e-54ce-4b5f-806b-e0f4eeb0d018","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:37:45.020577 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:45.020551    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:37:45.020691 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:45.020599    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:37:45.236233 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:45.236208    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7" event={"ID":"62eff799-47b9-474a-966f-2689ab4f1c17","Type":"ContainerStarted","Data":"31f600fa97d623a945fc8fc3c19dd1588b9f08f44d142b9c247dd73494326eaf"}
Oct 30 18:37:45.238104 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:45.238078    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tnkkl" event={"ID":"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c","Type":"ContainerStarted","Data":"ff113c2615ff52ac32e4260e5d6f3d0e32a205ed87032290dac88cad8337f235"}
Oct 30 18:37:45.239940 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:45.239903    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-26flh" event={"ID":"6b567074-99fd-4062-9fbe-a356eb37fe26","Type":"ContainerStarted","Data":"8aea5abe3fe58012a6786f31978ba967814dafb27c825502cab15cb72e2f7bcb"}
Oct 30 18:37:45.369081 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:45.369016    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q4mnk" podStartSLOduration=19.841309546 podStartE2EDuration="25.368998219s" totalImagesPullingTime="5.527688673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:43.346766262 +0000 UTC m=+44.846769878" watchObservedRunningTime="2026-10-30 18:37:45.368998219 +0000 UTC m=+46.869001836"
Oct 30 18:37:45.505700 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:45.505622    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tnkkl" podStartSLOduration=17.32044282 podStartE2EDuration="25.505605725s" totalImagesPullingTime="8.185162905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:45.369399394 +0000 UTC m=+46.869403010" watchObservedRunningTime="2026-10-30 18:37:45.505605725 +0000 UTC m=+47.005609340"
Oct 30 18:37:45.505700 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:45.505693    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-26flh" podStartSLOduration=17.284676417 podStartE2EDuration="25.505689305s" totalImagesPullingTime="8.221012888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:45.504410466 +0000 UTC m=+47.004414082" watchObservedRunningTime="2026-10-30 18:37:45.505689305 +0000 UTC m=+47.005692919"
Oct 30 18:37:46.056062 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:46.056019    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:46.056506 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:46.056071    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:46.056506 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:46.056087    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:46.056506 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:46.056198    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:46.056948 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:46.056511    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:46.056948 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:46.056658    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:46.244659 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:46.244629    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="519e082cbb3f5bc89a6849e00c43808e8ad238b4db9218f6f238346f7fa77951" exitCode=0
Oct 30 18:37:46.244771 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:46.244747    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfgck" event={"ID":"cb8aaa1a-7cea-437a-a436-09a8af560781","Type":"ContainerDied","Data":"519e082cbb3f5bc89a6849e00c43808e8ad238b4db9218f6f238346f7fa77951"}
Oct 30 18:37:46.374822 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:46.374795    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:46.892976 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:46.892946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:47.248499 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:47.248472    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hhdgg" event={"ID":"ed3bc506-2acc-4de7-8d28-738da0eaa6ee","Type":"ContainerStarted","Data":"80feba5f4cb007301d9fe598d5892b4f29cdb981ee4b8f5718c1e865dc45e89d"}
Oct 30 18:37:47.250368 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:47.250337    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ckgwd" event={"ID":"25811b4f-a869-49f0-ac40-b0f37274e97d","Type":"ContainerStarted","Data":"a24daf540fc808ef0f49f0c5768c9cfd64dcc2689ea69a0a565705d7bcf17f99"}
Oct 30 18:37:47.253128 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:47.253107    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7" event={"ID":"62eff799-47b9-474a-966f-2689ab4f1c17","Type":"ContainerStarted","Data":"dc804d1fee6b11896bb168e707c1f9e7d0db8c920e1a7f7f55b46666bc88d347"}
Oct 30 18:37:47.374694 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:47.374644    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hhdgg" podStartSLOduration=19.000306275 podStartE2EDuration="27.374627642s" totalImagesPullingTime="8.374321367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:47.374147121 +0000 UTC m=+48.874150737" watchObservedRunningTime="2026-10-30 18:37:47.374627642 +0000 UTC m=+48.874631259"
Oct 30 18:37:47.502762 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:47.502664    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7" podStartSLOduration=18.103671869 podStartE2EDuration="27.502650216s" totalImagesPullingTime="9.398978347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:47.502098824 +0000 UTC m=+49.002102443" watchObservedRunningTime="2026-10-30 18:37:47.502650216 +0000 UTC m=+49.002653831"
Oct 30 18:37:47.625855 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:47.625805    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ckgwd" podStartSLOduration=17.28620911 podStartE2EDuration="27.625792307s" totalImagesPullingTime="10.339583197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:47.625135754 +0000 UTC m=+49.125139371" watchObservedRunningTime="2026-10-30 18:37:47.625792307 +0000 UTC m=+49.125795922"
Oct 30 18:37:47.918791 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:47.918717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:48.056130 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:48.056102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:48.056130 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:48.056125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:48.056345 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:48.056132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:48.056345 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:48.056240    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:48.056452 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:48.056357    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:48.056501 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:48.056457    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:48.258099 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:48.258029    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="13345faca2eec3982227d267c1af4fe8168e7dd642339485f8212afa27c986fc" exitCode=0
Oct 30 18:37:48.258505 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:48.258132    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfgck" event={"ID":"cb8aaa1a-7cea-437a-a436-09a8af560781","Type":"ContainerDied","Data":"13345faca2eec3982227d267c1af4fe8168e7dd642339485f8212afa27c986fc"}
Oct 30 18:37:48.583544 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:48.583491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:49.174262 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:49.174221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:50.056045 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:50.055874    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:50.056439 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:50.055941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:50.056439 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:50.056132    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:50.056439 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:50.056198    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:50.056439 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:50.055948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:50.056439 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:50.056272    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:50.265579 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:50.265544    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="92ae65d9d2dac3f54e38a66d029667dc7fc00907d7389c6075337cf13ea30a25" exitCode=0
Oct 30 18:37:50.265756 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:50.265604    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfgck" event={"ID":"cb8aaa1a-7cea-437a-a436-09a8af560781","Type":"ContainerDied","Data":"92ae65d9d2dac3f54e38a66d029667dc7fc00907d7389c6075337cf13ea30a25"}
Oct 30 18:37:51.270175 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:51.269935    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2" event={"ID":"8bc74796-475a-4fe8-9113-a6dacb274acd","Type":"ContainerStarted","Data":"06d15b82f2c2aff13f9a68640bd4cdd3f9e2aa2d21c564d240d41a6a32b9e2f3"}
Oct 30 18:37:51.270877 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:51.270190    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2" event={"ID":"8bc74796-475a-4fe8-9113-a6dacb274acd","Type":"ContainerStarted","Data":"c7b2101ccd4e1fc7a3975ca2f79c5e83afde4fdc34a5d18acc6cdcba202a0a25"}
Oct 30 18:37:51.270877 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:51.270238    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2" event={"ID":"8bc74796-475a-4fe8-9113-a6dacb274acd","Type":"ContainerStarted","Data":"9209bdfc1f3272f8c9d40f14ca6bb97b1b8182678b29961a4e38ebe22f3473be"}
Oct 30 18:37:51.270877 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:51.270278    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2" event={"ID":"8bc74796-475a-4fe8-9113-a6dacb274acd","Type":"ContainerStarted","Data":"a761964fc0cab41e6b977613853d75947aafdf3fe8bdb038bbf4f6aa4a85d5a6"}
Oct 30 18:37:51.270877 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:51.270317    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2" event={"ID":"8bc74796-475a-4fe8-9113-a6dacb274acd","Type":"ContainerStarted","Data":"821b37688a5c85a1c8e1f9ae526553376b12c7812fef8f88454d61295e2ab4be"}
Oct 30 18:37:51.270877 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:51.270357    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2" event={"ID":"8bc74796-475a-4fe8-9113-a6dacb274acd","Type":"ContainerStarted","Data":"e62ae04e4afaa606986a9316394a428965e1d83d3d3b83339476addd3497ac1a"}
Oct 30 18:37:51.871821 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:51.871787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:51.872049 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:51.871843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:51.872049 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:51.871947    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:51.872049 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:51.871987    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:51.872049 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:51.872006    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs podName:e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:07.871989958 +0000 UTC m=+69.371993552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs") pod "network-metrics-daemon-w8jdh" (UID: "e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:37:51.872049 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:51.872030    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret podName:b81698fa-f5d9-48b0-85fa-29e71a85c441 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:07.872014939 +0000 UTC m=+69.372018548 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret") pod "global-pull-secret-syncer-mqdgz" (UID: "b81698fa-f5d9-48b0-85fa-29e71a85c441") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:37:51.972390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:51.972359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxv8\" (UniqueName: \"kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8\") pod \"network-check-target-v7pb2\" (UID: \"de6f539e-a438-42bb-98e6-d83b5926143a\") " pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:51.972642 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:51.972553    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:37:51.972642 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:51.972580    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:37:51.972642 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:51.972592    2578 projected.go:196] Error preparing data for projected volume kube-api-access-sgxv8 for pod openshift-network-diagnostics/network-check-target-v7pb2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:51.972763 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:51.972651    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8 podName:de6f539e-a438-42bb-98e6-d83b5926143a nodeName:}" failed. No retries permitted until 2026-10-30 18:38:07.972634114 +0000 UTC m=+69.472637720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sgxv8" (UniqueName: "kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8") pod "network-check-target-v7pb2" (UID: "de6f539e-a438-42bb-98e6-d83b5926143a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:37:52.056379 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:52.056352    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:52.056506 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:52.056428    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:52.056586 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:52.056511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:52.056586 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:52.056513    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:52.056679 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:52.056626    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:52.056717 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:52.056688    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:52.850823 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:52.850775    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:53.276931 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:53.276900    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2" event={"ID":"8bc74796-475a-4fe8-9113-a6dacb274acd","Type":"ContainerStarted","Data":"52f854078b734e3a3e3dbb105c3b57cbf5adb3f857b43296f42a273e9a2dd9cf"}
Oct 30 18:37:53.279696 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:53.279673    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="bd9ff1a7a3454acc085cb0ff83a9724539183298c174b1a32817380cfd5c974d" exitCode=0
Oct 30 18:37:53.279784 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:53.279705    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfgck" event={"ID":"cb8aaa1a-7cea-437a-a436-09a8af560781","Type":"ContainerDied","Data":"bd9ff1a7a3454acc085cb0ff83a9724539183298c174b1a32817380cfd5c974d"}
Oct 30 18:37:53.370402 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:53.370380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:54.056019 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:54.055991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:54.056405 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:54.056078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:54.056405 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:54.056092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:54.056405 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:54.056165    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:54.056405 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:54.056240    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:54.056405 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:54.056304    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:54.287095 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:54.287051    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="97d6d5a8a29ce3ebb802a44ce977f57e54278fe86a2e20b580374797b66f4cd7" exitCode=0
Oct 30 18:37:54.287245 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:54.287118    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfgck" event={"ID":"cb8aaa1a-7cea-437a-a436-09a8af560781","Type":"ContainerDied","Data":"97d6d5a8a29ce3ebb802a44ce977f57e54278fe86a2e20b580374797b66f4cd7"}
Oct 30 18:37:55.296929 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:55.296898    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfgck" event={"ID":"cb8aaa1a-7cea-437a-a436-09a8af560781","Type":"ContainerStarted","Data":"2705465fbc804e765a9b9733cf91c73972c271cbe85b6118656747c5919c75e1"}
Oct 30 18:37:55.484976 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:55.484887    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nfgck" podStartSLOduration=23.119438782 podStartE2EDuration="35.484872604s" totalImagesPullingTime="12.365433822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:55.484224801 +0000 UTC m=+56.984228416" watchObservedRunningTime="2026-10-30 18:37:55.484872604 +0000 UTC m=+56.984876219"
Oct 30 18:37:55.833540 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:55.833506    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:56.056588 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:56.056557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:56.056718 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:56.056636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:56.056718 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:56.056658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:56.056790 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:56.056763    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:56.056913 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:56.056892    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:56.056981 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:56.056957    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:56.272808 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:56.272742    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:56.303430 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:56.303400    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2" event={"ID":"8bc74796-475a-4fe8-9113-a6dacb274acd","Type":"ContainerStarted","Data":"8694fa3a05370fe5a5d485bc292f741e89663b721ceaf7969f03c9ebaf8bef1a"}
Oct 30 18:37:56.584830 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:56.584629    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2" podStartSLOduration=22.36653019 podStartE2EDuration="36.584614828s" totalImagesPullingTime="14.218084638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:56.584502889 +0000 UTC m=+58.084506504" watchObservedRunningTime="2026-10-30 18:37:56.584614828 +0000 UTC m=+58.084618437"
Oct 30 18:37:57.306218 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:57.306184    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:57.306720 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:57.306270    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:57.306720 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:57.306328    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:57.323181 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:57.323154    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:57.323473 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:57.323280    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:37:57.365049 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:57.365027    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:57.755390 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:57.755357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:58.055832 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:58.055769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:58.055960 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:58.055846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:58.055960 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:58.055928    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:58.055960 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:58.055944    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:58.056069 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:58.055988    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:37:58.056104 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:58.056064    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:58.130838 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:58.130815    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mqdgz"]
Oct 30 18:37:58.133854 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:58.133828    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v7pb2"]
Oct 30 18:37:58.134411 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:58.134391    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w8jdh"]
Oct 30 18:37:58.307773 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:58.307728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:37:58.308248 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:58.307728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:37:58.308248 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:58.307817    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:37:58.308248 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:37:58.307733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:37:58.308248 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:58.308200    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:37:58.308400 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:37:58.308288    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:38:00.056029 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:00.056003    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:00.056029 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:00.056021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:38:00.056551 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:00.056100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:38:00.056551 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:38:00.056107    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:38:00.056551 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:38:00.056202    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:38:00.056551 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:38:00.056261    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:38:01.523777 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:01.523745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:02.026452 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:02.026416    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:02.056589 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:02.056569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:02.056708 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:02.056590    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:38:02.056708 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:02.056602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:38:02.056708 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:38:02.056683    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqdgz" podUID="b81698fa-f5d9-48b0-85fa-29e71a85c441"
Oct 30 18:38:02.056865 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:38:02.056810    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v7pb2" podUID="de6f539e-a438-42bb-98e6-d83b5926143a"
Oct 30 18:38:02.056916 ip-10-0-129-31 kubenswrapper[2578]: E1030 18:38:02.056876    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8jdh" podUID="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1"
Oct 30 18:38:03.683432 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.683404    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeReady"
Oct 30 18:38:03.683776 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.683557    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:38:03.855478 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.855444    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bjg6d"]
Oct 30 18:38:03.858985 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.858970    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5hwl2"]
Oct 30 18:38:03.859126 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.859109    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:03.864834 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.864816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:03.914183 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.914158    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5hwl2"]
Oct 30 18:38:03.914776 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.914755    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bjg6d"]
Oct 30 18:38:03.917228 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.917211    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:38:03.917459 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.917442    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:38:03.917519 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.917472    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:38:03.918411 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.918393    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5fxhm\""
Oct 30 18:38:03.918411 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.918408    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:38:03.920287 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.920268    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4k96h\""
Oct 30 18:38:03.960874 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.960828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33db8810-0fe5-4b49-8136-231a566b46e5-crio-socket\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:03.960874 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.960858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33db8810-0fe5-4b49-8136-231a566b46e5-data-volume\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:03.961012 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.960884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d48e73c-4c04-452e-976d-cc74ef960a3a-config-volume\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:03.961012 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.960919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33db8810-0fe5-4b49-8136-231a566b46e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:03.961012 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.960968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8qw9\" (UniqueName: \"kubernetes.io/projected/33db8810-0fe5-4b49-8136-231a566b46e5-kube-api-access-r8qw9\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:03.961012 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.960998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhz66\" (UniqueName: \"kubernetes.io/projected/1d48e73c-4c04-452e-976d-cc74ef960a3a-kube-api-access-jhz66\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:03.961151 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.961024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d48e73c-4c04-452e-976d-cc74ef960a3a-metrics-tls\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:03.961151 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.961062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d48e73c-4c04-452e-976d-cc74ef960a3a-tmp-dir\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:03.961151 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.961111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33db8810-0fe5-4b49-8136-231a566b46e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:03.964926 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.964910    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tkcfx"]
Oct 30 18:38:03.971935 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.971918    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:03.992764 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:03.992745    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tkcfx"]
Oct 30 18:38:04.021388 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.021369    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:38:04.023522 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.023498    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:38:04.026977 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.026963    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:38:04.027051 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.027016    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:38:04.027433 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.027421    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:38:04.032803 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.032787    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kxkbv\""
Oct 30 18:38:04.055925 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.055908    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:38:04.056011 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.055938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:38:04.056011 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.055939    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:04.061370 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdnd\" (UniqueName: \"kubernetes.io/projected/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-kube-api-access-lmdnd\") pod \"ingress-canary-tkcfx\" (UID: \"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153\") " pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:04.061463 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33db8810-0fe5-4b49-8136-231a566b46e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:04.061769 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33db8810-0fe5-4b49-8136-231a566b46e5-crio-socket\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:04.061769 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33db8810-0fe5-4b49-8136-231a566b46e5-data-volume\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:04.061769 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d48e73c-4c04-452e-976d-cc74ef960a3a-config-volume\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:04.061769 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33db8810-0fe5-4b49-8136-231a566b46e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:04.061769 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8qw9\" (UniqueName: \"kubernetes.io/projected/33db8810-0fe5-4b49-8136-231a566b46e5-kube-api-access-r8qw9\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:04.061769 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhz66\" (UniqueName: \"kubernetes.io/projected/1d48e73c-4c04-452e-976d-cc74ef960a3a-kube-api-access-jhz66\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:04.061769 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-cert\") pod \"ingress-canary-tkcfx\" (UID: \"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153\") " pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:04.061769 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d48e73c-4c04-452e-976d-cc74ef960a3a-metrics-tls\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:04.061769 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d48e73c-4c04-452e-976d-cc74ef960a3a-tmp-dir\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:04.062160 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.061907    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33db8810-0fe5-4b49-8136-231a566b46e5-crio-socket\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:04.062160 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.062103    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33db8810-0fe5-4b49-8136-231a566b46e5-data-volume\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:04.062160 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.062137    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33db8810-0fe5-4b49-8136-231a566b46e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:04.062627 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.062608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d48e73c-4c04-452e-976d-cc74ef960a3a-tmp-dir\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:04.062700 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.062667    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d48e73c-4c04-452e-976d-cc74ef960a3a-config-volume\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:04.065402 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.065384    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d48e73c-4c04-452e-976d-cc74ef960a3a-metrics-tls\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:04.065562 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.065522    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33db8810-0fe5-4b49-8136-231a566b46e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:04.116301 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.116282    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:38:04.116391 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.116337    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:38:04.128117 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.128095    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hd9r2\""
Oct 30 18:38:04.128448 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.128431    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:38:04.128448 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.128443    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mrt5r\""
Oct 30 18:38:04.128594 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.128579    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kp722\""
Oct 30 18:38:04.128645 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.128610    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:38:04.131972 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.131951    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8qw9\" (UniqueName: \"kubernetes.io/projected/33db8810-0fe5-4b49-8136-231a566b46e5-kube-api-access-r8qw9\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:04.132401 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.132384    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhz66\" (UniqueName: \"kubernetes.io/projected/1d48e73c-4c04-452e-976d-cc74ef960a3a-kube-api-access-jhz66\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:04.162405 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.162386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-cert\") pod \"ingress-canary-tkcfx\" (UID: \"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153\") " pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:04.162497 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.162430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdnd\" (UniqueName: \"kubernetes.io/projected/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-kube-api-access-lmdnd\") pod \"ingress-canary-tkcfx\" (UID: \"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153\") " pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:04.164487 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.164471    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-cert\") pod \"ingress-canary-tkcfx\" (UID: \"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153\") " pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:04.181358 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.181337    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:04.190491 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.190470    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:04.227234 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.227209    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdnd\" (UniqueName: \"kubernetes.io/projected/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-kube-api-access-lmdnd\") pod \"ingress-canary-tkcfx\" (UID: \"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153\") " pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:04.290941 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.290911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:04.693770 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.693671    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5hwl2"]
Oct 30 18:38:04.696133 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:38:04.696110    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d48e73c_4c04_452e_976d_cc74ef960a3a.slice/crio-a2cf559b50c63326899545f021a9e75c82e74fdac6731a7ce49898fbec5348f3 WatchSource:0}: Error finding container a2cf559b50c63326899545f021a9e75c82e74fdac6731a7ce49898fbec5348f3: Status 404 returned error can't find the container with id a2cf559b50c63326899545f021a9e75c82e74fdac6731a7ce49898fbec5348f3
Oct 30 18:38:04.746283 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.746263    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tkcfx"]
Oct 30 18:38:04.748395 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:04.748375    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bjg6d"]
Oct 30 18:38:04.749305 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:38:04.749215    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adb3bc7_4601_4f2c_bce1_52f9ddd3d153.slice/crio-ddaae8806787630124a25b519d4ccc2d8e21792e6109ff125f52aa1eb6ef5b94 WatchSource:0}: Error finding container ddaae8806787630124a25b519d4ccc2d8e21792e6109ff125f52aa1eb6ef5b94: Status 404 returned error can't find the container with id ddaae8806787630124a25b519d4ccc2d8e21792e6109ff125f52aa1eb6ef5b94
Oct 30 18:38:04.750994 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:38:04.750973    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33db8810_0fe5_4b49_8136_231a566b46e5.slice/crio-180b6269ff9682bbd0f86b5cd4f3771ef9e6580d8fce4e10285bd922b8514d08 WatchSource:0}: Error finding container 180b6269ff9682bbd0f86b5cd4f3771ef9e6580d8fce4e10285bd922b8514d08: Status 404 returned error can't find the container with id 180b6269ff9682bbd0f86b5cd4f3771ef9e6580d8fce4e10285bd922b8514d08
Oct 30 18:38:05.329322 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:05.329153    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5hwl2" event={"ID":"1d48e73c-4c04-452e-976d-cc74ef960a3a","Type":"ContainerStarted","Data":"a2cf559b50c63326899545f021a9e75c82e74fdac6731a7ce49898fbec5348f3"}
Oct 30 18:38:05.330651 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:05.330627    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjg6d" event={"ID":"33db8810-0fe5-4b49-8136-231a566b46e5","Type":"ContainerStarted","Data":"4ac16eeb003933f979c167e1446715f12af963e7eed373ad1ff1111c3d8123bc"}
Oct 30 18:38:05.330765 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:05.330677    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjg6d" event={"ID":"33db8810-0fe5-4b49-8136-231a566b46e5","Type":"ContainerStarted","Data":"180b6269ff9682bbd0f86b5cd4f3771ef9e6580d8fce4e10285bd922b8514d08"}
Oct 30 18:38:05.331631 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:05.331608    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tkcfx" event={"ID":"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153","Type":"ContainerStarted","Data":"ddaae8806787630124a25b519d4ccc2d8e21792e6109ff125f52aa1eb6ef5b94"}
Oct 30 18:38:06.336803 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:06.336765    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjg6d" event={"ID":"33db8810-0fe5-4b49-8136-231a566b46e5","Type":"ContainerStarted","Data":"4d30462056bf2d80680168a8359a0d880cf852492a0ae4e2dc919dece819090e"}
Oct 30 18:38:06.365948 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:06.365884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:06.707805 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:06.707724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:07.342624 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:07.341964    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5hwl2" event={"ID":"1d48e73c-4c04-452e-976d-cc74ef960a3a","Type":"ContainerStarted","Data":"61e1fdc027fb1f2d2078a48755cb964d57b5c7f0c743a4c6fa9b15b53a8d7dde"}
Oct 30 18:38:07.342624 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:07.342026    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5hwl2" event={"ID":"1d48e73c-4c04-452e-976d-cc74ef960a3a","Type":"ContainerStarted","Data":"7316d018ec233ec3de55c24f781eb784c22112a1bacf10be0ce07f88ca8c78e4"}
Oct 30 18:38:07.342624 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:07.342099    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:07.504587 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:07.504515    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-5hwl2" podStartSLOduration=2.1486414 podStartE2EDuration="4.504497546s" totalImagesPullingTime="2.355856146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:07.502955719 +0000 UTC m=+69.002959336" watchObservedRunningTime="2026-10-30 18:38:07.504497546 +0000 UTC m=+69.004501164"
Oct 30 18:38:07.891002 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:07.890975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:07.891151 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:07.891071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:38:07.894380 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:07.894356    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:38:07.894657 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:07.894639    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:07.991985 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:07.991902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxv8\" (UniqueName: \"kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8\") pod \"network-check-target-v7pb2\" (UID: \"de6f539e-a438-42bb-98e6-d83b5926143a\") " pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:38:07.994907 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:07.994881    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgxv8\" (UniqueName: \"kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8\") pod \"network-check-target-v7pb2\" (UID: \"de6f539e-a438-42bb-98e6-d83b5926143a\") " pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:38:08.000499 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.000467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:38:08.011714 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.011687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:08.030556 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.030503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:38:08.176372 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.176344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:08.346796 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.346762    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjg6d" event={"ID":"33db8810-0fe5-4b49-8136-231a566b46e5","Type":"ContainerStarted","Data":"fe98600fd25b23ef3064a2659d13dbf5589708a264d0ffb249d32383db777884"}
Oct 30 18:38:08.348322 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.348298    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tkcfx" event={"ID":"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153","Type":"ContainerStarted","Data":"87be021df88605a08ff0743cc910693ae3e85180075806a945b8c0af571cebab"}
Oct 30 18:38:08.385706 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.385685    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w8jdh"]
Oct 30 18:38:08.425479 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:38:08.425456    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c06abd_1f83_4cd9_8eb6_e6efbbbc17c1.slice/crio-dc05edfa5d99f124eb7a6dfa795084ef53c46b95d2ca98539b1cbe56716ae198 WatchSource:0}: Error finding container dc05edfa5d99f124eb7a6dfa795084ef53c46b95d2ca98539b1cbe56716ae198: Status 404 returned error can't find the container with id dc05edfa5d99f124eb7a6dfa795084ef53c46b95d2ca98539b1cbe56716ae198
Oct 30 18:38:08.437292 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.437270    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mqdgz"]
Oct 30 18:38:08.438206 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.438123    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v7pb2"]
Oct 30 18:38:08.439828 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:38:08.439807    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81698fa_f5d9_48b0_85fa_29e71a85c441.slice/crio-dec6015928ed16ed0f5f59a791823c604e81315f4b155bf5e3159985d47ccb02 WatchSource:0}: Error finding container dec6015928ed16ed0f5f59a791823c604e81315f4b155bf5e3159985d47ccb02: Status 404 returned error can't find the container with id dec6015928ed16ed0f5f59a791823c604e81315f4b155bf5e3159985d47ccb02
Oct 30 18:38:08.440384 ip-10-0-129-31 kubenswrapper[2578]: W1030 18:38:08.440357    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6f539e_a438_42bb_98e6_d83b5926143a.slice/crio-a56e859b6d8765ddb91c2a09343a6c2b1f701379a6a32f9e74280510ad5306ab WatchSource:0}: Error finding container a56e859b6d8765ddb91c2a09343a6c2b1f701379a6a32f9e74280510ad5306ab: Status 404 returned error can't find the container with id a56e859b6d8765ddb91c2a09343a6c2b1f701379a6a32f9e74280510ad5306ab
Oct 30 18:38:08.469856 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.469816    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bjg6d" podStartSLOduration=2.171163709 podStartE2EDuration="5.469806065s" totalImagesPullingTime="3.298642356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:38:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:08.469001181 +0000 UTC m=+69.969004795" watchObservedRunningTime="2026-10-30 18:38:08.469806065 +0000 UTC m=+69.969809671"
Oct 30 18:38:08.476293 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.476270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:08.598954 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:08.598918    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tkcfx" podStartSLOduration=2.540892236 podStartE2EDuration="5.598907785s" totalImagesPullingTime="3.058015549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:08.597836909 +0000 UTC m=+70.097840524" watchObservedRunningTime="2026-10-30 18:38:08.598907785 +0000 UTC m=+70.098911401"
Oct 30 18:38:09.351861 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:09.351830    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8jdh" event={"ID":"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1","Type":"ContainerStarted","Data":"dc05edfa5d99f124eb7a6dfa795084ef53c46b95d2ca98539b1cbe56716ae198"}
Oct 30 18:38:09.352883 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:09.352857    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v7pb2" event={"ID":"de6f539e-a438-42bb-98e6-d83b5926143a","Type":"ContainerStarted","Data":"a56e859b6d8765ddb91c2a09343a6c2b1f701379a6a32f9e74280510ad5306ab"}
Oct 30 18:38:09.353823 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:09.353805    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mqdgz" event={"ID":"b81698fa-f5d9-48b0-85fa-29e71a85c441","Type":"ContainerStarted","Data":"dec6015928ed16ed0f5f59a791823c604e81315f4b155bf5e3159985d47ccb02"}
Oct 30 18:38:10.361061 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:10.361021    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8jdh" event={"ID":"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1","Type":"ContainerStarted","Data":"6536358ccc27a2bbe08d614813dc65b13418eb892db3af8b992649a959c965ad"}
Oct 30 18:38:10.361508 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:10.361088    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8jdh" event={"ID":"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1","Type":"ContainerStarted","Data":"903fc003a52a2021466f7dc9700d9882ef637fb1f67dba5e8f50c7e37db2ba99"}
Oct 30 18:38:10.485819 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:10.485763    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w8jdh" podStartSLOduration=48.864217125 podStartE2EDuration="50.48574414s" totalImagesPullingTime="1.621527015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:10.484994338 +0000 UTC m=+71.984997956" watchObservedRunningTime="2026-10-30 18:38:10.48574414 +0000 UTC m=+71.985747758"
Oct 30 18:38:10.899122 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:10.899070    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:11.478626 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:11.478588    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:12.370282 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:12.370245    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v7pb2" event={"ID":"de6f539e-a438-42bb-98e6-d83b5926143a","Type":"ContainerStarted","Data":"5df9694fc30989be5971d25c878b80ec47273c6def944d3100217b8e65293376"}
Oct 30 18:38:12.370455 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:12.370413    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:38:12.494117 ip-10-0-129-31 kubenswrapper[2578]: I1030 18:38:12.494033    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v7pb2" podStartSLOduration=48.742135214 podStartE2EDuration="52.494020344s" totalImagesPullingTime="3.75188513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:12.493480084 +0000 UTC m=+73.993483701" watchObservedRunningTime="2026-10-30 18:38:12.494020344 +0000 UTC m=+73.994023959"
Oct 30 18:38:13.760971 ip-10-0-129-31 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:38:13.779016 ip-10-0-129-31 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:38:13.779187 ip-10-0-129-31 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:38:13.779345 ip-10-0-129-31 systemd[1]: kubelet.service: Consumed 3.002s CPU time.
Oct 30 18:38:13.787174 ip-10-0-129-31 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:38:13.861276 ip-10-0-129-31 kubenswrapper[4853]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:13.861276 ip-10-0-129-31 kubenswrapper[4853]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:38:13.861276 ip-10-0-129-31 kubenswrapper[4853]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:13.861276 ip-10-0-129-31 kubenswrapper[4853]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:38:13.863364 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863246    4853 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:13.863364 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863364    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863374    4853 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863378    4853 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863383    4853 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863388    4853 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863391    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863394    4853 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863397    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863399    4853 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863402    4853 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863405    4853 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863408    4853 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863411    4853 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863423    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863426    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863428    4853 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863431    4853 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863434    4853 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863436    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:13.863468 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863439    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863442    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863444    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863447    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863449    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863452    4853 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863455    4853 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863457    4853 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863460    4853 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863463    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863466    4853 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863469    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863471    4853 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863474    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863477    4853 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863480    4853 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863482    4853 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863485    4853 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863487    4853 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863546    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:13.864037 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863551    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863554    4853 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863558    4853 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863562    4853 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863568    4853 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863574    4853 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863577    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863581    4853 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863585    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863589    4853 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863593    4853 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863597    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863603    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863608    4853 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863616    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863619    4853 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863622    4853 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863625    4853 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863628    4853 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863631    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:13.864550 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863633    4853 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863636    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863639    4853 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863641    4853 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863644    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863646    4853 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863649    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863651    4853 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863653    4853 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863656    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863659    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863661    4853 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863664    4853 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863666    4853 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863669    4853 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863671    4853 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863674    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863680    4853 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863683    4853 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863686    4853 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:13.865055 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863688    4853 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863692    4853 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863696    4853 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863699    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863701    4853 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863704    4853 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863707    4853 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863710    4853 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863713    4853 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863715    4853 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863718    4853 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863720    4853 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863723    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863726    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863728    4853 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863731    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863733    4853 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863736    4853 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863738    4853 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:13.865601 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863741    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863744    4853 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863748    4853 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863750    4853 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863753    4853 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863755    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.863758    4853 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864504    4853 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864512    4853 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864514    4853 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864520    4853 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864522    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864550    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864555    4853 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864557    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864560    4853 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864563    4853 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864566    4853 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864568    4853 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864571    4853 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:13.866070 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864574    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864576    4853 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864579    4853 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864582    4853 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864585    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864588    4853 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864590    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864593    4853 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864595    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864598    4853 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864601    4853 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864603    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864606    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864608    4853 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864611    4853 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864613    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864616    4853 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864618    4853 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864620    4853 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864623    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:13.866611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864626    4853 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864628    4853 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864633    4853 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864635    4853 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864638    4853 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864642    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864644    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864647    4853 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864649    4853 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864652    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864656    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864658    4853 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864661    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864663    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864666    4853 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864668    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864671    4853 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864673    4853 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864676    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864679    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:13.867122 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864681    4853 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864683    4853 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864686    4853 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864688    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864691    4853 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864693    4853 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864696    4853 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864698    4853 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864700    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864703    4853 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864705    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864710    4853 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864713    4853 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864716    4853 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864720    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864723    4853 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864725    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864728    4853 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864730    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:13.867647 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864733    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864736    4853 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864738    4853 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864741    4853 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864743    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864746    4853 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864748    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864751    4853 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864754    4853 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864756    4853 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864759    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864761    4853 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864764    4853 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864766    4853 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864769    4853 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864771    4853 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864774    4853 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864776    4853 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864779    4853 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864781    4853 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:13.868095 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864784    4853 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864787    4853 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864789    4853 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864792    4853 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864794    4853 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864797    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864800    4853 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864805    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864808    4853 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864810    4853 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864813    4853 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864816    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864820    4853 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.864822    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865288    4853 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865293    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865296    4853 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865299    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865302    4853 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:13.868597 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865305    4853 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865308    4853 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865312    4853 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865316    4853 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865319    4853 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865322    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865325    4853 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865327    4853 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865330    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865332    4853 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865335    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865337    4853 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865340    4853 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865342    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865345    4853 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865348    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865351    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865354    4853 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865356    4853 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:13.869144 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865359    4853 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865362    4853 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865364    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865367    4853 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865369    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865372    4853 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865375    4853 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865377    4853 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865380    4853 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865383    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865385    4853 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865388    4853 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865390    4853 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865393    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865395    4853 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865398    4853 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865400    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865403    4853 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865405    4853 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865407    4853 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:13.869614 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865410    4853 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865413    4853 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865415    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865419    4853 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865421    4853 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865424    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865426    4853 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865429    4853 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865431    4853 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865434    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865436    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865439    4853 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865441    4853 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865444    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865446    4853 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865448    4853 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865451    4853 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865453    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865457    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:13.870128 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865460    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865463    4853 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865465    4853 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865468    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865470    4853 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865473    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865475    4853 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865478    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865480    4853 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865483    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865485    4853 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865487    4853 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865490    4853 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865492    4853 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865495    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865497    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865500    4853 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865502    4853 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865505    4853 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:13.870611 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865508    4853 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865512    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865514    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865517    4853 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865519    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865522    4853 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865540    4853 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865543    4853 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865545    4853 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865548    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865551    4853 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865553    4853 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865557    4853 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865560    4853 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865563    4853 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865566    4853 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865568    4853 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865571    4853 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865573    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865576    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:13.871081 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865578    4853 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865582    4853 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865585    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.865587    4853 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865693    4853 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865711    4853 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865719    4853 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865728    4853 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865734    4853 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865739    4853 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865745    4853 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865749    4853 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865752    4853 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865754    4853 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865758    4853 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865761    4853 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865764    4853 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865767    4853 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865770    4853 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865773    4853 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865775    4853 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865778    4853 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865784    4853 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865787    4853 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:38:13.871615 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865791    4853 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865794    4853 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865797    4853 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865801    4853 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865805    4853 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865807    4853 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865811    4853 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865814    4853 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865816    4853 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865819    4853 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865823    4853 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865826    4853 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865830    4853 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865833    4853 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865836    4853 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865839    4853 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865841    4853 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865844    4853 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865848    4853 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865851    4853 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865854    4853 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865857    4853 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865860    4853 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865864    4853 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:38:13.872205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865867    4853 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865870    4853 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865873    4853 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865876    4853 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865879    4853 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865881    4853 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865886    4853 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865889    4853 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865892    4853 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865895    4853 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865899    4853 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865902    4853 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865905    4853 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865908    4853 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865911    4853 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865914    4853 flags.go:64] FLAG: --help="false"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865917    4853 flags.go:64] FLAG: --hostname-override="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865920    4853 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865923    4853 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865926    4853 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865930    4853 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865933    4853 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865936    4853 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:38:13.872845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865938    4853 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865941    4853 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865944    4853 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865947    4853 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865950    4853 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865953    4853 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865955    4853 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865958    4853 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865961    4853 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865964    4853 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865966    4853 flags.go:64] FLAG: --lock-file=""
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865969    4853 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865972    4853 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865975    4853 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865980    4853 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865984    4853 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865987    4853 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865990    4853 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865993    4853 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865996    4853 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.865999    4853 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866002    4853 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866006    4853 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866009    4853 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866013    4853 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:38:13.873411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866017    4853 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866020    4853 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866022    4853 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866025    4853 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866028    4853 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866031    4853 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866033    4853 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866040    4853 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866043    4853 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866046    4853 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866049    4853 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866052    4853 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866054    4853 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866057    4853 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866060    4853 flags.go:64] FLAG: --port="10250"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866063    4853 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866066    4853 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ff983347bfafa4b2"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866069    4853 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866072    4853 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866075    4853 flags.go:64] FLAG: --register-node="true"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866078    4853 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866082    4853 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866086    4853 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866089    4853 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866092    4853 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866095    4853 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:38:13.874256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866098    4853 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866101    4853 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866104    4853 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866107    4853 flags.go:64] FLAG: --runonce="false"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866110    4853 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866113    4853 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866116    4853 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866119    4853 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866122    4853 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866125    4853 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866128    4853 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866131    4853 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866133    4853 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866136    4853 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866139    4853 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866141    4853 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866149    4853 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866152    4853 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866155    4853 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866160    4853 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866163    4853 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866166    4853 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866169    4853 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866172    4853 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866175    4853 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:38:13.875125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866178    4853 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866180    4853 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866183    4853 flags.go:64] FLAG: --v="2"
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866188    4853 flags.go:64] FLAG: --version="false"
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866193    4853 flags.go:64] FLAG: --vmodule=""
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866196    4853 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866200    4853 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866348    4853 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866352    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866355    4853 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866358    4853 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866360    4853 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866363    4853 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866366    4853 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866369    4853 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866371    4853 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866374    4853 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866377    4853 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866380    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866382    4853 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866385    4853 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:13.875795 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866387    4853 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866390    4853 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866394    4853 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866397    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866399    4853 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866402    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866405    4853 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866407    4853 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866410    4853 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866414    4853 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866417    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866421    4853 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866424    4853 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866426    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866430    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866433    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866435    4853 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866438    4853 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866440    4853 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866443    4853 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:13.876313 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866445    4853 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866448    4853 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866450    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866453    4853 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866456    4853 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866458    4853 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866461    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866463    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866466    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866469    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866471    4853 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866474    4853 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866476    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866479    4853 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866482    4853 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866484    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866487    4853 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866489    4853 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866492    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:13.876917 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866494    4853 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866497    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866499    4853 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866502    4853 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866504    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866508    4853 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866511    4853 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866515    4853 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866518    4853 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866521    4853 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866537    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866540    4853 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866543    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866545    4853 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866548    4853 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866550    4853 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866553    4853 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866559    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866562    4853 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:13.877516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866565    4853 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866567    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866570    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866572    4853 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866575    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866578    4853 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866580    4853 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866583    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866586    4853 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866589    4853 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866591    4853 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866594    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866597    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866599    4853 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866602    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866604    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866607    4853 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866609    4853 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866612    4853 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:13.877997 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866614    4853 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866619    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866622    4853 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866625    4853 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866627    4853 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866630    4853 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866632    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866635    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866637    4853 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866640    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866642    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866645    4853 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866648    4853 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866653    4853 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866656    4853 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.866683    4853 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:13.878491 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866724    4853 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866728    4853 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866731    4853 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866734    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866737    4853 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866741    4853 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866743    4853 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866746    4853 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866749    4853 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866751    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866754    4853 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866756    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866759    4853 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866761    4853 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866764    4853 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866768    4853 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866772    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866775    4853 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866779    4853 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866782    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:13.878956 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866785    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866787    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866790    4853 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866792    4853 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866795    4853 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866797    4853 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866800    4853 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866802    4853 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866805    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866807    4853 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866815    4853 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866818    4853 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866821    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866823    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866826    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866828    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866831    4853 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866835    4853 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866837    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:13.879516 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866840    4853 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866842    4853 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866845    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866853    4853 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866856    4853 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866858    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866861    4853 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866863    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866866    4853 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866869    4853 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866871    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866877    4853 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866880    4853 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866883    4853 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866885    4853 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866888    4853 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866890    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866893    4853 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866895    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866898    4853 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:13.880007 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866900    4853 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866903    4853 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866905    4853 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866908    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866911    4853 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866913    4853 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866916    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866918    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866921    4853 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866923    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866926    4853 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866929    4853 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866932    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866934    4853 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866937    4853 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866939    4853 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866942    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866944    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866947    4853 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:13.880498 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866949    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866951    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866954    4853 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866956    4853 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866962    4853 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866964    4853 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866967    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866969    4853 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866972    4853 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866974    4853 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866977    4853 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866979    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866982    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866984    4853 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866987    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866989    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866992    4853 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866995    4853 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.866998    4853 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:13.880988 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.867001    4853 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.867003    4853 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.867005    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.867008    4853 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.867011    4853 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.867014    4853 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.867016    4853 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.867019    4853 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.867021    4853 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.870610    4853 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.870626    4853 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.870644    4853 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.870649    4853 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870702    4853 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870707    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870711    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870714    4853 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870717    4853 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870720    4853 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870723    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:13.881465 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870726    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870728    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870731    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870734    4853 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870736    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870739    4853 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870741    4853 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870744    4853 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870747    4853 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870750    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870752    4853 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870756    4853 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870760    4853 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870763    4853 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870766    4853 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870768    4853 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870771    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870774    4853 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870776    4853 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:13.881962 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870779    4853 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870781    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870784    4853 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870788    4853 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870792    4853 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870795    4853 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870797    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870800    4853 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870802    4853 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870805    4853 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870808    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870810    4853 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870812    4853 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870815    4853 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870817    4853 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870820    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870822    4853 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870825    4853 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:13.882415 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870827    4853 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870830    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870832    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870836    4853 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870838    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870841    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870843    4853 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870846    4853 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870848    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870851    4853 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870854    4853 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870856    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870859    4853 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870861    4853 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870864    4853 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870866    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870869    4853 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870871    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870875    4853 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:13.883003 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870878    4853 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870880    4853 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870882    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870885    4853 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870887    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870889    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870892    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870894    4853 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870897    4853 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870900    4853 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870902    4853 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870905    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870907    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870909    4853 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870912    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870914    4853 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870917    4853 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870920    4853 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870923    4853 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870925    4853 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:13.883466 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870928    4853 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870930    4853 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870933    4853 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870936    4853 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870938    4853 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870941    4853 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870943    4853 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870946    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870948    4853 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870951    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870953    4853 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870955    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870959    4853 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870961    4853 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870964    4853 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870966    4853 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870969    4853 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870971    4853 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870974    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870976    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:13.883963 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870979    4853 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870981    4853 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.870983    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.871010    4853 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871167    4853 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871172    4853 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871175    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871178    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871181    4853 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871184    4853 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871187    4853 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871189    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871192    4853 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871194    4853 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871197    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871200    4853 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:13.884488 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871202    4853 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871205    4853 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871207    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871210    4853 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871212    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871215    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871217    4853 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871221    4853 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871224    4853 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871228    4853 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871231    4853 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871233    4853 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871236    4853 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871238    4853 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871241    4853 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871243    4853 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871246    4853 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871249    4853 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871251    4853 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:13.884890 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871253    4853 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871256    4853 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871258    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871261    4853 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871263    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871266    4853 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871268    4853 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871270    4853 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871273    4853 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871276    4853 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871278    4853 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871281    4853 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871283    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871286    4853 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871288    4853 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871291    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871294    4853 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871296    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871298    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871301    4853 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:13.885367 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871303    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871306    4853 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871308    4853 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871311    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871314    4853 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871316    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871318    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871321    4853 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871323    4853 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871326    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871328    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871331    4853 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871333    4853 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871335    4853 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871338    4853 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871340    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871343    4853 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871345    4853 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871348    4853 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871350    4853 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:13.885879 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871352    4853 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871355    4853 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871357    4853 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871360    4853 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871363    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871366    4853 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871368    4853 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871371    4853 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871373    4853 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871376    4853 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871378    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871380    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871383    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871385    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871388    4853 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871390    4853 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871395    4853 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871398    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871400    4853 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:13.886353 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871402    4853 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871405    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871407    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871409    4853 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871412    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871415    4853 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871418    4853 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871422    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871425    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871429    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871432    4853 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871435    4853 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871437    4853 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871440    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871442    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:13.886915 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871445    4853 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.871467    4853 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871496    4853 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871499    4853 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871502    4853 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871505    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871507    4853 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871510    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871512    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871514    4853 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871518    4853 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871521    4853 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871539    4853 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871544    4853 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871547    4853 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:38:13.887285 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871550    4853 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871553    4853 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871555    4853 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871557    4853 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871560    4853 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871563    4853 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871565    4853 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871568    4853 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871570    4853 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871573    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871575    4853 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871579    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871582    4853 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871584    4853 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871587    4853 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871590    4853 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871592    4853 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871595    4853 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871597    4853 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871599    4853 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:38:13.887748 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871602    4853 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871604    4853 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871607    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871610    4853 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871612    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871614    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871617    4853 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871619    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871622    4853 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871625    4853 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871627    4853 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871630    4853 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871632    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871635    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871638    4853 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871640    4853 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871643    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871645    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871648    4853 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871650    4853 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:38:13.888274 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871653    4853 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871655    4853 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871658    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871661    4853 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871663    4853 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871666    4853 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871668    4853 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871670    4853 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871673    4853 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871675    4853 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871678    4853 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871680    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871682    4853 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871685    4853 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871687    4853 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871690    4853 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871692    4853 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871694    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871699    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:38:13.888884 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871702    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871704    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871707    4853 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871709    4853 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871712    4853 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871714    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871716    4853 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871719    4853 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871722    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871725    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871727    4853 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871730    4853 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871732    4853 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871737    4853 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871740    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871743    4853 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871746    4853 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871749    4853 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871751    4853 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:38:13.889617 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871754    4853 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871756    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871759    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871761    4853 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871763    4853 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871766    4853 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871768    4853 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871771    4853 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871773    4853 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871776    4853 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871778    4853 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871781    4853 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871784    4853 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871787    4853 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:38:13.871789    4853 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.871855    4853 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.872743    4853 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.872787    4853 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.872965    4853 server.go:1014] "Starting client certificate rotation"
Oct 30 18:38:13.890172 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.873104    4853 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:38:13.890980 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.873159    4853 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:32:17 +0000 UTC" deadline="2028-04-06 11:42:51.661332897 +0000 UTC"
Oct 30 18:38:13.890980 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.873199    4853 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12569h4m37.788135951s"
Oct 30 18:38:13.890980 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.873381    4853 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:38:13.890980 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.875504    4853 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:38:13.890980 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.876843    4853 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:38:13.890980 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.878401    4853 log.go:25] "Validated CRI v1 image API"
Oct 30 18:38:13.890980 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.879162    4853 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:38:13.890980 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.880686    4853 fs.go:135] Filesystem UUIDs: map[0411990e-163a-415d-87eb-de467208efee:/dev/nvme0n1p3 67ce03ed-9ba8-4c2d-995e-c834f89d14c0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 18:38:13.891345 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.880706    4853 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/180b6269ff9682bbd0f86b5cd4f3771ef9e6580d8fce4e10285bd922b8514d08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/180b6269ff9682bbd0f86b5cd4f3771ef9e6580d8fce4e10285bd922b8514d08/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18b2914f029dcd2d7d184b647181af4988040700ee0d3989d5daa4996aa5b4da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18b2914f029dcd2d7d184b647181af4988040700ee0d3989d5daa4996aa5b4da/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3158b9688a04e6027f1c4e6594c2a025483d8bae4bb9dcac1c86b6982e7084f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3158b9688a04e6027f1c4e6594c2a025483d8bae4bb9dcac1c86b6982e7084f7/userdata/shm major:0 minor:104 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/398b4da201df26917b4d3f8cd1286a2c955b47eb1e754b955c106996dba51846/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/398b4da201df26917b4d3f8cd1286a2c955b47eb1e754b955c106996dba51846/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/642a234072bbe9e5423e488fd3d5b99fcd9d9b8aebdd0385cadebbde7b2a3b77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/642a234072bbe9e5423e488fd3d5b99fcd9d9b8aebdd0385cadebbde7b2a3b77/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71011d241d3eef21a5ef163b81b0569dd94f02594b79ea8c7027bc267cf11c16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71011d241d3eef21a5ef163b81b0569dd94f02594b79ea8c7027bc267cf11c16/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/866ea6c26969fd3288b62075097aa4e24eead5dcdb9c1c8194822a8fe46b646f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/866ea6c26969fd3288b62075097aa4e24eead5dcdb9c1c8194822a8fe46b646f/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ce11b8efa0c0a4d42e9503ad491db65e2ea21a568c781e349cc65819153bd2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ce11b8efa0c0a4d42e9503ad491db65e2ea21a568c781e349cc65819153bd2f/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2cf559b50c63326899545f021a9e75c82e74fdac6731a7ce49898fbec5348f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2cf559b50c63326899545f021a9e75c82e74fdac6731a7ce49898fbec5348f3/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a56e859b6d8765ddb91c2a09343a6c2b1f701379a6a32f9e74280510ad5306ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a56e859b6d8765ddb91c2a09343a6c2b1f701379a6a32f9e74280510ad5306ab/userdata/shm major:0 minor:269 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bab21f8676dafd9bbdd1d1c7f2ede6953d0b9dbe26ca1b7275fdfeed42856b8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bab21f8676dafd9bbdd1d1c7f2ede6953d0b9dbe26ca1b7275fdfeed42856b8f/userdata/shm major:0 minor:109 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc05edfa5d99f124eb7a6dfa795084ef53c46b95d2ca98539b1cbe56716ae198/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc05edfa5d99f124eb7a6dfa795084ef53c46b95d2ca98539b1cbe56716ae198/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ddaae8806787630124a25b519d4ccc2d8e21792e6109ff125f52aa1eb6ef5b94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ddaae8806787630124a25b519d4ccc2d8e21792e6109ff125f52aa1eb6ef5b94/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dec6015928ed16ed0f5f59a791823c604e81315f4b155bf5e3159985d47ccb02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dec6015928ed16ed0f5f59a791823c604e81315f4b155bf5e3159985d47ccb02/userdata/shm major:0 minor:267 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ede44013528faa80e66e24b7e962fb69a7f308c016ac52540affa302fabfdb69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ede44013528faa80e66e24b7e962fb69a7f308c016ac52540affa302fabfdb69/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f67508545cfba1c53d9bee04cb2bbd24cd4db60605f77f023f2f210e040eb33d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f67508545cfba1c53d9bee04cb2bbd24cd4db60605f77f023f2f210e040eb33d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d48e73c-4c04-452e-976d-cc74ef960a3a/volumes/kubernetes.io~projected/kube-api-access-jhz66:{mountpoint:/var/lib/kubelet/pods/1d48e73c-4c04-452e-976d-cc74ef960a3a/volumes/kubernetes.io~projected/kube-api-access-jhz66 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d48e73c-4c04-452e-976d-cc74ef960a3a/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1d48e73c-4c04-452e-976d-cc74ef960a3a/volumes/kubernetes.io~secret/metrics-tls major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25811b4f-a869-49f0-ac40-b0f37274e97d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/25811b4f-a869-49f0-ac40-b0f37274e97d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25811b4f-a869-49f0-ac40-b0f37274e97d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/25811b4f-a869-49f0-ac40-b0f37274e97d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25811b4f-a869-49f0-ac40-b0f37274e97d/volumes/kubernetes.io~projected/kube-api-access-zk6ls:{mountpoint:/var/lib/kubelet/pods/25811b4f-a869-49f0-ac40-b0f37274e97d/volumes/kubernetes.io~projected/kube-api-access-zk6ls major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bd81706-d26f-420f-a97d-270ebbc96057/volumes/kubernetes.io~projected/kube-api-access-lggv5:{mountpoint:/var/lib/kubelet/pods/2bd81706-d26f-420f-a97d-270ebbc96057/volumes/kubernetes.io~projected/kube-api-access-lggv5 major:0 minor:71 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bd81706-d26f-420f-a97d-270ebbc96057/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2bd81706-d26f-420f-a97d-270ebbc96057/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bd81706-d26f-420f-a97d-270ebbc96057/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/2bd81706-d26f-420f-a97d-270ebbc96057/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33db8810-0fe5-4b49-8136-231a566b46e5/volumes/kubernetes.io~projected/kube-api-access-r8qw9:{mountpoint:/var/lib/kubelet/pods/33db8810-0fe5-4b49-8136-231a566b46e5/volumes/kubernetes.io~projected/kube-api-access-r8qw9 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33db8810-0fe5-4b49-8136-231a566b46e5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/33db8810-0fe5-4b49-8136-231a566b46e5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47d77dc5-6269-4949-b4fa-60c617073f7c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/47d77dc5-6269-4949-b4fa-60c617073f7c/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153/volumes/kubernetes.io~projected/kube-api-access-lmdnd:{mountpoint:/var/lib/kubelet/pods/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153/volumes/kubernetes.io~projected/kube-api-access-lmdnd major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153/volumes/kubernetes.io~secret/cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62eff799-47b9-474a-966f-2689ab4f1c17/volumes/kubernetes.io~projected/kube-api-access-ffxdt:{mountpoint:/var/lib/kubelet/pods/62eff799-47b9-474a-966f-2689ab4f1c17/volumes/kubernetes.io~projected/kube-api-access-ffxdt major:0 minor:73 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b567074-99fd-4062-9fbe-a356eb37fe26/volumes/kubernetes.io~projected/kube-api-access-6xwf7:{mountpoint:/var/lib/kubelet/pods/6b567074-99fd-4062-9fbe-a356eb37fe26/volumes/kubernetes.io~projected/kube-api-access-6xwf7 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bc74796-475a-4fe8-9113-a6dacb274acd/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8bc74796-475a-4fe8-9113-a6dacb274acd/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bc74796-475a-4fe8-9113-a6dacb274acd/volumes/kubernetes.io~projected/kube-api-access-lshdb:{mountpoint:/var/lib/kubelet/pods/8bc74796-475a-4fe8-9113-a6dacb274acd/volumes/kubernetes.io~projected/kube-api-access-lshdb major:0 minor:72 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bc74796-475a-4fe8-9113-a6dacb274acd/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8bc74796-475a-4fe8-9113-a6dacb274acd/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b81698fa-f5d9-48b0-85fa-29e71a85c441/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b81698fa-f5d9-48b0-85fa-29e71a85c441/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c/volumes/kubernetes.io~projected/kube-api-access-dfpk7:{mountpoint:/var/lib/kubelet/pods/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c/volumes/kubernetes.io~projected/kube-api-access-dfpk7 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb8aaa1a-7cea-437a-a436-09a8af560781/volumes/kubernetes.io~projected/kube-api-access-cf42g:{mountpoint:/var/lib/kubelet/pods/cb8aaa1a-7cea-437a-a436-09a8af560781/volumes/kubernetes.io~projected/kube-api-access-cf42g major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de6f539e-a438-42bb-98e6-d83b5926143a/volumes/kubernetes.io~projected/kube-api-access-sgxv8:{mountpoint:/var/lib/kubelet/pods/de6f539e-a438-42bb-98e6-d83b5926143a/volumes/kubernetes.io~projected/kube-api-access-sgxv8 major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27/volumes/kubernetes.io~projected/kube-api-access-74dqb:{mountpoint:/var/lib/kubelet/pods/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27/volumes/kubernetes.io~projected/kube-api-access-74dqb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1/volumes/kubernetes.io~projected/kube-api-access-7jkqx:{mountpoint:/var/lib/kubelet/pods/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1/volumes/kubernetes.io~projected/kube-api-access-7jkqx major:0 minor:74 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1/volumes/kubernetes.io~secret/metrics-certs major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed3bc506-2acc-4de7-8d28-738da0eaa6ee/volumes/kubernetes.io~projected/kube-api-access-j7zhh:{mountpoint:/var/lib/kubelet/pods/ed3bc506-2acc-4de7-8d28-738da0eaa6ee/volumes/kubernetes.io~projected/kube-api-access-j7zhh major:0 minor:68 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-101:{mountpoint:/var/lib/containers/storage/overlay/0e41db89ee8eaaf54c7e52123a54670d84e60678b2d4bd4b5927ece471bd3c34/merged major:0 minor:101 fsType:overlay blockSize:0} overlay_0-103:{mountpoint:/var/lib/containers/storage/overlay/528d68d5aa1334771603f4747552c191aabe77832cc0b5bdcff2d74bc9a9e6fe/merged major:0 minor:103 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/deb47f6ecd237bea953c60cb23ea4f4ed77950c520c3bd080b96dc0e6442f6f1/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/b90ca2e65dfba5b3902ef38508e2b136c3dbb786be08adb4f6045ff1390e8e6a/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/5178314856bf3b7869bfb0c24bf8f51aaa7c98cffc07c2bf8683577ecf8a1f13/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-115:{mountpoint:/var/lib/containers/storage/overlay/b0a0830c674507b240345b7899a6d3df86072ed36588e96f8e7fe0e9f988dcda/merged major:0 minor:115 fsType:overlay blockSize:0} overlay_0-120:{mountpoint:/var/lib/containers/storage/overlay/963c6d97ac43eb817f0618a18a064ce6a5a75ec80fb1c6ba210b8813c04e5a9f/merged major:0 minor:120 fsType:overlay blockSize:0} overlay_0-125:{mountpoint:/var/lib/containers/storage/overlay/e411fcb377b2af425c1234a12fa8a5a913eabfe7217a133fb2089749d0f720ed/merged major:0 minor:125 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/33340d72d20cbf68222d11b2005d1a463e9a473d50a18df10d19cc2ae10fc9f0/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/8f3ea8b01094fa80317161554f2d7bd9bb622a57be715204046ac44b14b7b884/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/3fb42f334c5fcf609f9dd10a4b72801af92a043bd4f1d53fa7c85f8a9e477f25/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/c994e7f0af4dca8c60fcc7b527c1cca4272d12f02f610b971980c914f116f67d/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/c8c280efc41a55684edd97064910153cd932ece19b0a3e4a33c1b8bdb6172f73/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/34e96c3584f807f9c9f356bfd9ad08fc28ba643030d826128e3e98b2d71d7136/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-160:{mountpoint:/var/lib/containers/storage/overlay/9660ddc973f7533c3275bc16e85bf1df478f5f0ad198d7855a2b8339d7f21106/merged major:0 minor:160 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bdba78e3ee638eed4d5d446a1e8635bc80fa2af0f2722c0342fb78789f752042/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-168:{mountpoint:/var/lib/containers/storage/overlay/88d16432e3bd93ced9513b4b347ae89a1d3a35e6a4640fb6b831dfcfbb756d05/merged major:0 minor:168 fsType:overlay blockSize:0} overlay_0-173:{mountpoint:/var/lib/containers/storage/overlay/edcadf1f9b8ef0a4638c896dec7a70f36778d2c5ea39e2a6c2e5eac631e1e474/merged major:0 minor:173 fsType:overlay blockSize:0} overlay_0-178:{mountpoint:/var/lib/containers/storage/overlay/548114d298f03da2d2d86e332696759319d344c63af30d6a13583c55096a1706/merged major:0 minor:178 fsType:overlay blockSize:0} overlay_0-183:{mountpoint:/var/lib/containers/storage/overlay/76f2b765ff86d4605af5362d0c033f684f0941791f9d47528de18d7e4d7d3a6c/merged major:0 minor:183 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/55e47c7c377790ad7a350e2e7946d40e98fbba3aa4b5d103bab49a196dae3793/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-193:{mountpoint:/var/lib/containers/storage/overlay/dd83aa5acf33eb1f58ffaca8edfd0ef462a3e8c0df657a1c0a8ab98db4ef3e5c/merged major:0 minor:193 fsType:overlay blockSize:0} overlay_0-198:{mountpoint:/var/lib/containers/storage/overlay/fca774d666f4b19c1de988bb11e9a3e72a651b89018009a4955709ad08bf0d31/merged major:0 minor:198 fsType:overlay blockSize:0} overlay_0-203:{mountpoint:/var/lib/containers/storage/overlay/86225e81fbb8ba2482e84034541d35bf312fac6a083a12bea5a483fd9fcb4199/merged major:0 minor:203 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/6a6b5684f4f08820f30f09faab0719a3c249b07083566ee7a1118dfef68227d3/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/a0d284bedec0a5e868edccfcf0dc3392976ed4892b79036262f060e6a07622df/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/8ad5bd2e4f84bab1a140f54fd45709c31d5dcaa6d1b2c8cac1378d4877a31cb9/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/6c15b8f92a36cf5ee465f428ac3c857898bdc874aa5df047410dd624cde4ad3e/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/8d242effd098c719dc65515548efe8c475aa7694c807e6b5acf581d5011649f0/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/4ba5a50572810a3d5c53674ab44bcec59cf503cf8e288929203b353664cefff7/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/bda41837334af2bd9a3963becee2fa0fadee857a581b82ccbbae5d99814b403e/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/2891f8f2353f4cc8341367751b11c832953d1738edfc97291cc6c98589b1e492/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/de63ad4a96a92f7cc8c36c3ceff24072d348ffa72aa669c98926ce4cdfc555d3/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/6b6d5c81d0252fdd54520e3d7093a5f2a488869e3e6205ee75ad262d98d1d3a2/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/96cbf225f6e34315be8d428075839110596502ea102d5222349835dd7abfa792/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/c8b242689bfa5afb6cd944e70b3a43c14f8d93fec5305ab6ef274b9c04901e15/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/da3f679b36b913621c10ed78b72cccb4f341fcb159323baa52515f07701a71d9/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/9251766a9ee9e66e4fd07beefc0bb00826ede58bd9b7ac19421fe6a49150d1c7/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/74de01aed5dc38ed6b08b0d751b36ad9855a8bc69ece5ed71f1b281a6768fd4c/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/96d65e6f638e2608f19d5d0ce7935d29126688dfcd4dc3d47c5175fedb734cd3/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/8460f8f8b252927bc58668e80cdbe8a6f2c55e05a046dd7aa876c1828b22bb82/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/27b1864a77957666073e3539e7e2bfdc29bf9443d2ce66363407e25d9a2479e7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/44b18f9f3f71f3b13f48f52761a0022dd45ca142be2a94a3994b29d27bc273bd/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-55:{mountpoint:/var/lib/containers/storage/overlay/87f5bbaeffab1bba0ddecf8c8cdc16c1bc3c1515619e989676ce0f33a91b3c55/merged major:0 minor:55 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/d884b9f4f2abae6b407f3739078cca46e9eec143ac6b07eb10c101df882babef/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/e60b222d4fde213ab905179f39bdc8432c68ebe8781b166d6a567d2a32d5ce0f/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/dee7747bee5a54ca4119b88870dc664d9746ad2df8ea8a8b112a7ebc030c71bd/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/de8654f0ea7da0ac8a2cbfc33c5d163c52aec8637eec57f77005bb3ff2b38b0e/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/cd6974ecfff4802dcf7f7c5b2be54f735aae6aec8978ce24f7ac307ee0356aa6/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/127d850e614ad2bbf8d68fcbe7662e11788eff900e4a22c1a245e78dffb3c96f/merged major:0 minor:95 fsType:overlay blockSize:0}]
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.886877    4853 manager.go:217] Machine: {Timestamp:2026-10-30 18:38:13.886379994 +0000 UTC m=+0.056736832 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109048 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a0beaa8721ca71972feb5eae7b9ed SystemUUID:ec2a0bea-a872-1ca7-1972-feb5eae7b9ed BootID:4cab329c-1a0c-4033-a1ff-40f2e26f657f Filesystems:[{Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/180b6269ff9682bbd0f86b5cd4f3771ef9e6580d8fce4e10285bd922b8514d08/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a56e859b6d8765ddb91c2a09343a6c2b1f701379a6a32f9e74280510ad5306ab/userdata/shm DeviceMajor:0 DeviceMinor:269 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bab21f8676dafd9bbdd1d1c7f2ede6953d0b9dbe26ca1b7275fdfeed42856b8f/userdata/shm DeviceMajor:0 DeviceMinor:109 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153/volumes/kubernetes.io~projected/kube-api-access-lmdnd DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dec6015928ed16ed0f5f59a791823c604e81315f4b155bf5e3159985d47ccb02/userdata/shm DeviceMajor:0 DeviceMinor:267 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-55 DeviceMajor:0 DeviceMinor:55 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb8aaa1a-7cea-437a-a436-09a8af560781/volumes/kubernetes.io~projected/kube-api-access-cf42g DeviceMajor:0 DeviceMinor:69 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2bd81706-d26f-420f-a97d-270ebbc96057/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8bc74796-475a-4fe8-9113-a6dacb274acd/volumes/kubernetes.io~projected/kube-api-access-lshdb DeviceMajor:0 DeviceMinor:72 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ce11b8efa0c0a4d42e9503ad491db65e2ea21a568c781e349cc65819153bd2f/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-183 DeviceMajor:0 DeviceMinor:183 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bc74796-475a-4fe8-9113-a6dacb274acd/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/33db8810-0fe5-4b49-8136-231a566b46e5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ddaae8806787630124a25b519d4ccc2d8e21792e6109ff125f52aa1eb6ef5b94/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:263 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3158b9688a04e6027f1c4e6594c2a025483d8bae4bb9dcac1c86b6982e7084f7/userdata/shm DeviceMajor:0 DeviceMinor:104 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-193 DeviceMajor:0 DeviceMinor:193 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2cf559b50c63326899545f021a9e75c82e74fdac6731a7ce49898fbec5348f3/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2bd81706-d26f-420f-a97d-270ebbc96057/volumes/kubernetes.io~projected/kube-api-access-lggv5 DeviceMajor:0 DeviceMinor:71 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/62eff799-47b9-474a-966f-2689ab4f1c17/volumes/kubernetes.io~projected/kube-api-access-ffxdt DeviceMajor:0 DeviceMinor:73 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-101 DeviceMajor:0 DeviceMinor:101 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b81698fa-f5d9-48b0-85fa-29e71a85c441/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:258 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25811b4f-a869-49f0-ac40-b0f37274e97d/volumes/kubernetes.io~projected/kube-api-access-zk6ls DeviceMajor:0 DeviceMinor:70 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/398b4da201df26917b4d3f8cd1286a2c955b47eb1e754b955c106996dba51846/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27/volumes/kubernetes.io~projected/kube-api-access-74dqb DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1/volumes/kubernetes.io~projected/kube-api-access-7jkqx DeviceMajor:0 DeviceMinor:74 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/866ea6c26969fd3288b62075097aa4e24eead5dcdb9c1c8194822a8fe46b646f/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/47d77dc5-6269-4949-b4fa-60c617073f7c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-203 DeviceMajor:0 DeviceMinor:203 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d48e73c-4c04-452e-976d-cc74ef960a3a/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/de6f539e-a438-42bb-98e6-d83b5926143a/volumes/kubernetes.io~projected/kube-api-access-sgxv8 DeviceMajor:0 DeviceMinor:264 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2bd81706-d26f-420f-a97d-270ebbc96057/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71011d241d3eef21a5ef163b81b0569dd94f02594b79ea8c7027bc267cf11c16/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-120 DeviceMajor:0 DeviceMinor:120 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33db8810-0fe5-4b49-8136-231a566b46e5/volumes/kubernetes.io~projected/kube-api-access-r8qw9 DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1d48e73c-4c04-452e-976d-cc74ef960a3a/volumes/kubernetes.io~projected/kube-api-access-jhz66 DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-115 DeviceMajor:0 DeviceMinor:115 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-125 DeviceMajor:0 DeviceMinor:125 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-160 DeviceMajor:0 DeviceMinor:160 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25811b4f-a869-49f0-ac40-b0f37274e97d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c/volumes/kubernetes.io~projected/kube-api-access-dfpk7 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-178 DeviceMajor:0 DeviceMinor:178 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-198 DeviceMajor:0 DeviceMinor:198 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bc74796-475a-4fe8-9113-a6dacb274acd/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25811b4f-a869-49f0-ac40-b0f37274e97d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/642a234072bbe9e5423e488fd3d5b99fcd9d9b8aebdd0385cadebbde7b2a3b77/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-103 DeviceMajor:0 DeviceMinor:103 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-168 DeviceMajor:0 DeviceMinor:168 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-173 DeviceMajor:0 DeviceMinor:173 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc05edfa5d99f124eb7a6dfa795084ef53c46b95d2ca98539b1cbe56716ae198/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ede44013528faa80e66e24b7e962fb69a7f308c016ac52540affa302fabfdb69/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f67508545cfba1c53d9bee04cb2bbd24cd4db60605f77f023f2f210e040eb33d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18b2914f029dcd2d7d184b647181af4988040700ee0d3989d5daa4996aa5b4da/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b567074-99fd-4062-9fbe-a356eb37fe26/volumes/kubernetes.io~projected/kube-api-access-6xwf7 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ed3bc506-2acc-4de7-8d28-738da0eaa6ee/volumes/kubernetes.io~projected/kube-api-access-j7zhh DeviceMajor:0 DeviceMinor:68 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:180b6269ff9682b MacAddress:ba:66:92:24:06:5c Speed:10000 Mtu:8901} {Name:a2cf559b50c6332 MacAddress:0e:7f:5e:c7:a8:38 Speed:10000 Mtu:8901} {Name:a56e859b6d8765d MacAddress:3e:1b:b0:9f:5c:87 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ca:56:50:6f Speed:0 Mtu:9001} {Name:br-int MacAddress:16:71:f3:4d:1f:9f Speed:0 Mtu:8901} {Name:dc05edfa5d99f12 MacAddress:5a:ac:1c:e3:d3:b3 Speed:10000 Mtu:8901} {Name:ddaae8806787630 MacAddress:fa:1d:ec:bf:00:d2 Speed:10000 Mtu:8901} {Name:dec6015928ed16e MacAddress:ca:9b:99:bf:7f:7f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ca:56:50:6f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:02:e8:01:69:e8:25 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:49:dc:c6:21:0a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.887145    4853 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.887183    4853 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.887462    4853 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.887486    4853 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-31.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.887626    4853 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.887633    4853 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.887644    4853 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.887657    4853 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.887741    4853 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.887902    4853 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.888012    4853 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.888043    4853 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.888052    4853 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.888061    4853 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.888068    4853 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.888076    4853 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.888733    4853 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.888744    4853 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:38:13.891651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.889310    4853 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:38:13.892870 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.892854    4853 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:38:13.892870 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.892870    4853 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:38:13.892970 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.892960    4853 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:38:13.893145 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893134    4853 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:38:13.893176 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893163    4853 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:38:13.893253 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893243    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:38:13.893293 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893258    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:38:13.893293 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893264    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:38:13.893293 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893269    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:38:13.893293 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893275    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:38:13.893293 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893280    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:38:13.893293 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893285    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:38:13.893293 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893290    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:38:13.893293 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893296    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:38:13.893521 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893303    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:38:13.893521 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893311    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:38:13.893521 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893320    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:38:13.893521 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893342    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:38:13.893521 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.893348    4853 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:38:13.896477 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.896465    4853 server.go:1263] "Started kubelet"
Oct 30 18:38:13.896869 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.896541    4853 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:38:13.896869 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.896570    4853 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:38:13.896869 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.896641    4853 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:38:13.897066 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.896932    4853 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:38:13.897062 ip-10-0-129-31 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:38:13.898842 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.898828    4853 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:38:13.899710 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.899694    4853 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:38:13.900336 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.899824    4853 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:38:13.900336 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.899872    4853 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:32:36 +0000 UTC" deadline="2028-05-01 14:41:29.806076734 +0000 UTC"
Oct 30 18:38:13.900336 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.899904    4853 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13172h3m15.906175679s"
Oct 30 18:38:13.900336 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.899935    4853 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:38:13.900336 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.899942    4853 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:38:13.900336 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.900035    4853 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:38:13.900925 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:13.900610    4853 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:38:13.901885 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.901859    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8aaa1a-7cea-437a-a436-09a8af560781" volumeName="kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:38:13.901985 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.901888    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d48e73c-4c04-452e-976d-cc74ef960a3a" volumeName="kubernetes.io/secret/1d48e73c-4c04-452e-976d-cc74ef960a3a-metrics-tls" seLinuxMountContext=""
Oct 30 18:38:13.901985 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.901899    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd81706-d26f-420f-a97d-270ebbc96057" volumeName="kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 18:38:13.901985 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.901923    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33db8810-0fe5-4b49-8136-231a566b46e5" volumeName="kubernetes.io/secret/33db8810-0fe5-4b49-8136-231a566b46e5-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 18:38:13.901985 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.901938    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4adb3bc7-4601-4f2c-bce1-52f9ddd3d153" volumeName="kubernetes.io/projected/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-kube-api-access-lmdnd" seLinuxMountContext=""
Oct 30 18:38:13.901985 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.901956    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b567074-99fd-4062-9fbe-a356eb37fe26" volumeName="kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:38:13.901985 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.901963    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8aaa1a-7cea-437a-a436-09a8af560781" volumeName="kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:38:13.901985 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.901974    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd81706-d26f-420f-a97d-270ebbc96057" volumeName="kubernetes.io/empty-dir/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-textfile" seLinuxMountContext=""
Oct 30 18:38:13.901985 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.901985    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33db8810-0fe5-4b49-8136-231a566b46e5" volumeName="kubernetes.io/empty-dir/33db8810-0fe5-4b49-8136-231a566b46e5-data-volume" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.901993    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33db8810-0fe5-4b49-8136-231a566b46e5" volumeName="kubernetes.io/projected/33db8810-0fe5-4b49-8136-231a566b46e5-kube-api-access-r8qw9" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902009    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47d77dc5-6269-4949-b4fa-60c617073f7c" volumeName="kubernetes.io/configmap/47d77dc5-6269-4949-b4fa-60c617073f7c-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902016    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4adb3bc7-4601-4f2c-bce1-52f9ddd3d153" volumeName="kubernetes.io/secret/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-cert" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902023    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b567074-99fd-4062-9fbe-a356eb37fe26" volumeName="kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902035    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b567074-99fd-4062-9fbe-a356eb37fe26" volumeName="kubernetes.io/projected/6b567074-99fd-4062-9fbe-a356eb37fe26-kube-api-access-6xwf7" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902045    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c" volumeName="kubernetes.io/projected/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-kube-api-access-dfpk7" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902052    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d48e73c-4c04-452e-976d-cc74ef960a3a" volumeName="kubernetes.io/configmap/1d48e73c-4c04-452e-976d-cc74ef960a3a-config-volume" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902074    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33db8810-0fe5-4b49-8136-231a566b46e5" volumeName="kubernetes.io/configmap/33db8810-0fe5-4b49-8136-231a566b46e5-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902081    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc74796-475a-4fe8-9113-a6dacb274acd" volumeName="kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-config" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902094    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc74796-475a-4fe8-9113-a6dacb274acd" volumeName="kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-env-overrides" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902106    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c" volumeName="kubernetes.io/empty-dir/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-tmp-dir" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902114    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8aaa1a-7cea-437a-a436-09a8af560781" volumeName="kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902122    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6f539e-a438-42bb-98e6-d83b5926143a" volumeName="kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902145    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1" volumeName="kubernetes.io/projected/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-kube-api-access-7jkqx" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902157    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d48e73c-4c04-452e-976d-cc74ef960a3a" volumeName="kubernetes.io/empty-dir/1d48e73c-4c04-452e-976d-cc74ef960a3a-tmp-dir" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902175    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd81706-d26f-420f-a97d-270ebbc96057" volumeName="kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902187    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd81706-d26f-420f-a97d-270ebbc96057" volumeName="kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-tls" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902193    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62eff799-47b9-474a-966f-2689ab4f1c17" volumeName="kubernetes.io/projected/62eff799-47b9-474a-966f-2689ab4f1c17-kube-api-access-ffxdt" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902199    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23b1354-07f4-4b59-a6a1-a1c96c8c6b27" volumeName="kubernetes.io/configmap/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-serviceca" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902213    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1" volumeName="kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902220    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed3bc506-2acc-4de7-8d28-738da0eaa6ee" volumeName="kubernetes.io/projected/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-kube-api-access-j7zhh" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902238    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d48e73c-4c04-452e-976d-cc74ef960a3a" volumeName="kubernetes.io/projected/1d48e73c-4c04-452e-976d-cc74ef960a3a-kube-api-access-jhz66" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902255    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc74796-475a-4fe8-9113-a6dacb274acd" volumeName="kubernetes.io/secret/8bc74796-475a-4fe8-9113-a6dacb274acd-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902261    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25811b4f-a869-49f0-ac40-b0f37274e97d" volumeName="kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-tuned" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902278    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25811b4f-a869-49f0-ac40-b0f37274e97d" volumeName="kubernetes.io/projected/25811b4f-a869-49f0-ac40-b0f37274e97d-kube-api-access-zk6ls" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902294    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed3bc506-2acc-4de7-8d28-738da0eaa6ee" volumeName="kubernetes.io/configmap/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902304    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25811b4f-a869-49f0-ac40-b0f37274e97d" volumeName="kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-tmp" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902323    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd81706-d26f-420f-a97d-270ebbc96057" volumeName="kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-metrics-client-ca" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902335    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc74796-475a-4fe8-9113-a6dacb274acd" volumeName="kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902341    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b81698fa-f5d9-48b0-85fa-29e71a85c441" volumeName="kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902353    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8aaa1a-7cea-437a-a436-09a8af560781" volumeName="kubernetes.io/projected/cb8aaa1a-7cea-437a-a436-09a8af560781-kube-api-access-cf42g" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902390    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23b1354-07f4-4b59-a6a1-a1c96c8c6b27" volumeName="kubernetes.io/projected/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-kube-api-access-74dqb" seLinuxMountContext=""
Oct 30 18:38:13.902374 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902397    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bd81706-d26f-420f-a97d-270ebbc96057" volumeName="kubernetes.io/projected/2bd81706-d26f-420f-a97d-270ebbc96057-kube-api-access-lggv5" seLinuxMountContext=""
Oct 30 18:38:13.904356 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902404    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47d77dc5-6269-4949-b4fa-60c617073f7c" volumeName="kubernetes.io/secret/47d77dc5-6269-4949-b4fa-60c617073f7c-agent-certs" seLinuxMountContext=""
Oct 30 18:38:13.904356 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902416    4853 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc74796-475a-4fe8-9113-a6dacb274acd" volumeName="kubernetes.io/projected/8bc74796-475a-4fe8-9113-a6dacb274acd-kube-api-access-lshdb" seLinuxMountContext=""
Oct 30 18:38:13.904356 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902423    4853 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:38:13.904356 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.902428    4853 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:38:13.904356 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.903656    4853 factory.go:55] Registering systemd factory
Oct 30 18:38:13.904356 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.903670    4853 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:38:13.904759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.904743    4853 factory.go:153] Registering CRI-O factory
Oct 30 18:38:13.904823 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.904762    4853 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:38:13.904823 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.904811    4853 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:38:13.904918 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.904829    4853 factory.go:103] Registering Raw factory
Oct 30 18:38:13.904918 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.904843    4853 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:38:13.905207 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.905193    4853 manager.go:319] Starting recovery of all containers
Oct 30 18:38:13.917652 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.917619    4853 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:38:13.932918 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.932895    4853 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:38:13.932918 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.932920    4853 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:38:13.933076 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.932943    4853 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:38:13.933076 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:13.932983    4853 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:38:13.938479 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.938453    4853 generic.go:356] "Generic (PLEG): container finished" podID="e1c96e16bc130f55ab259bba0b1cc606" containerID="167cbe67b9c7afdfff5902fce31928cc091cb6900361b21f8442ba10eb4cd531" exitCode=0
Oct 30 18:38:13.950362 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.950341    4853 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="97d6d5a8a29ce3ebb802a44ce977f57e54278fe86a2e20b580374797b66f4cd7" exitCode=0
Oct 30 18:38:13.950362 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.950360    4853 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="bd9ff1a7a3454acc085cb0ff83a9724539183298c174b1a32817380cfd5c974d" exitCode=0
Oct 30 18:38:13.950478 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.950367    4853 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="92ae65d9d2dac3f54e38a66d029667dc7fc00907d7389c6075337cf13ea30a25" exitCode=0
Oct 30 18:38:13.950478 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.950373    4853 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="13345faca2eec3982227d267c1af4fe8168e7dd642339485f8212afa27c986fc" exitCode=0
Oct 30 18:38:13.950478 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.950380    4853 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="519e082cbb3f5bc89a6849e00c43808e8ad238b4db9218f6f238346f7fa77951" exitCode=0
Oct 30 18:38:13.950478 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.950385    4853 generic.go:356] "Generic (PLEG): container finished" podID="cb8aaa1a-7cea-437a-a436-09a8af560781" containerID="8c692a768959ee16fed8810d4f88d32c87bbfe2edcc2b2112fbb0b1db6c5222f" exitCode=0
Oct 30 18:38:13.977009 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.976988    4853 generic.go:356] "Generic (PLEG): container finished" podID="2bd81706-d26f-420f-a97d-270ebbc96057" containerID="945aa1e8973a743ec68bcc66c0ee4dc6b4284a60f05e0aef86fa3be052ea4e77" exitCode=0
Oct 30 18:38:13.985061 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:13.985045    4853 manager.go:324] Recovery completed
Oct 30 18:38:14.000356 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.000340    4853 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:14.001406 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:14.001375    4853 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:38:14.002103 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.002089    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:14.002165 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.002119    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:14.002165 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.002131    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:14.003935 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.003917    4853 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:38:14.003935 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.003929    4853 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:14.004051 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.003930    4853 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:38:14.004051 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004020    4853 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:38:14.004136 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004122    4853 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:38:14.004181 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004135    4853 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:38:14.004181 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004150    4853 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:38:14.004181 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004155    4853 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:38:14.004181 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004160    4853 policy_none.go:50] "Start"
Oct 30 18:38:14.004181 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004168    4853 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:38:14.004181 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004176    4853 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:38:14.004440 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004287    4853 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:38:14.004440 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004293    4853 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:38:14.004440 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004298    4853 policy_none.go:44] "Start"
Oct 30 18:38:14.004778 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004762    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:14.004856 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004796    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:14.004856 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.004811    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:14.008919 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.008887    4853 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:38:14.012409 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.012365    4853 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:38:14.012409 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:14.012386    4853 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:38:14.012409 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.012395    4853 server.go:87] "Starting device plugin registration server"
Oct 30 18:38:14.012655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.012643    4853 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:38:14.012712 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.012654    4853 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:38:14.012779 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.012760    4853 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:38:14.012917 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.012873    4853 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:38:14.013002 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.012929    4853 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:38:14.013002 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.012939    4853 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:38:14.013099 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.013002    4853 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:38:14.012916861Z","UUID":"2c79bfe6-610e-484e-91e1-14457b846c67","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:38:14.014339 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.014322    4853 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:38:14.014435 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.014356    4853 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:38:14.014501 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:14.014486    4853 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:38:14.014579 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:14.014518    4853 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-31.us-west-2.compute.internal\" not found"
Oct 30 18:38:14.034448 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.033949    4853 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"]
Oct 30 18:38:14.034552 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.034521    4853 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:14.035845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.035822    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:14.035925 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.035878    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:14.035925 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.035889    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:14.036102 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.036092    4853 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:14.036221 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.036208    4853 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:14.036748 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.036734    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:14.036827 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.036760    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:14.036827 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.036762    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:14.036827 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.036773    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:14.036827 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.036782    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:14.036827 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.036792    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:14.036972 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:14.036887    4853 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.037005 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.036965    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"e1c96e16bc130f55ab259bba0b1cc606","Type":"ContainerDied","Data":"167cbe67b9c7afdfff5902fce31928cc091cb6900361b21f8442ba10eb4cd531"}
Oct 30 18:38:14.037005 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.036993    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"e1c96e16bc130f55ab259bba0b1cc606","Type":"ContainerStarted","Data":"436c037d7bb2fe39c100a157df3fe024333180ac982d2c8664a0a7b7cb35e685"}
Oct 30 18:38:14.037063 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.037050    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"38d71fab1c4baec8ab7a9648db13d15f","Type":"ContainerStarted","Data":"fef671a98bae65212702d6bb17f9dcf96ca27e78f3495f1d606023a31fb3ed96"}
Oct 30 18:38:14.037092 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.037062    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal" event={"ID":"38d71fab1c4baec8ab7a9648db13d15f","Type":"ContainerStarted","Data":"18b2914f029dcd2d7d184b647181af4988040700ee0d3989d5daa4996aa5b4da"}
Oct 30 18:38:14.037092 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.037073    4853 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:14.037693 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.037667    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:14.037752 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.037705    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:14.037752 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.037715    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:14.037826 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:14.037815    4853 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-31.us-west-2.compute.internal\" not found" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.087886 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.087862    4853 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:38:14.113501 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.113480    4853 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:38:14.114215 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.114200    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:38:14.114309 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.114288    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:38:14.114309 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.114311    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:38:14.114461 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.114348    4853 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.178761 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.178742    4853 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:14.179316 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.179276    4853 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:14.179407 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.179379    4853 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:14.179407 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.179395    4853 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:38:14.188900 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.188876    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.188986 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.188910    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.188986 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.188936    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d71fab1c4baec8ab7a9648db13d15f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"38d71fab1c4baec8ab7a9648db13d15f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.201011 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.200986    4853 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.233349 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.233324    4853 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.233445 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.233388    4853 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.233892 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.233870    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeNotReady"
Oct 30 18:38:14.233970 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.233898    4853 setters.go:546] "Node became not ready" node="ip-10-0-129-31.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T18:38:14Z","lastTransitionTime":"2026-10-30T18:38:14Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 18:38:14.269922 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.269869    4853 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:14.269922 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:14.269906    4853 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.269922 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.269917    4853 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.288650 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.288629    4853 nodeinfomanager.go:471] existing CSINode "ip-10-0-129-31.us-west-2.compute.internal" is owned by different node (oldNodeID="f0134497-0bb5-42d5-95a1-211919d7f74f", newNodeID=""), cleaning up...
Oct 30 18:38:14.289710 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.289695    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.289759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.289718    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d71fab1c4baec8ab7a9648db13d15f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"38d71fab1c4baec8ab7a9648db13d15f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.289759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.289736    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.289827 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.289769    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.289827 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.289802    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38d71fab1c4baec8ab7a9648db13d15f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"38d71fab1c4baec8ab7a9648db13d15f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.289889 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.289802    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c96e16bc130f55ab259bba0b1cc606-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" (UID: \"e1c96e16bc130f55ab259bba0b1cc606\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.373733 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.373712    4853 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:14.373811 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:14.373745    4853 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.888879 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.888857    4853 apiserver.go:52] "Watching apiserver"
Oct 30 18:38:14.980790 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.980768    4853 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.980920 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.980881    4853 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:14.998361 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.998340    4853 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:38:14.998810 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:14.998791    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-hhdgg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7","openshift-network-diagnostics/network-check-target-v7pb2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal","openshift-insights/insights-runtime-extractor-bjg6d","openshift-cluster-node-tuning-operator/tuned-ckgwd","openshift-image-registry/node-ca-q4mnk","openshift-monitoring/node-exporter-bjkm5","kube-system/global-pull-secret-syncer-mqdgz","openshift-dns/node-resolver-tnkkl","openshift-multus/multus-26flh","kube-system/konnectivity-agent-25n6m","kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-7k5m2","openshift-dns/dns-default-5hwl2","openshift-multus/network-metrics-daemon-w8jdh","openshift-ingress-canary/ingress-canary-tkcfx","openshift-multus/multus-additional-cni-plugins-nfgck"]
Oct 30 18:38:15.037917 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.037889    4853 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:15.038012 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:15.037934    4853 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:15.039558 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.039522    4853 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:38:15.039652 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:15.039572    4853 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 18:38:15.059060 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.059019    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:38:15.059060 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.059019    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.059249 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.059024    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:38:15.059301 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.059286    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.059349 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.059324    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.059349 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.059341    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.059617 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.059600    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:38:15.059748 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.059605    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.059748 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.059640    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:38:15.059748 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.059708    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thsk7\""
Oct 30 18:38:15.060138 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060117    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k6xhn\""
Oct 30 18:38:15.060138 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060121    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.060286 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060149    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.060286 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060173    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.060286 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060173    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:38:15.060286 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060192    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:38:15.060286 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060261    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.060286 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060279    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mrt5r\""
Oct 30 18:38:15.060498 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060311    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sslwz\""
Oct 30 18:38:15.060498 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060328    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.060498 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060355    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hd9r2\""
Oct 30 18:38:15.060498 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060409    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w6mkd\""
Oct 30 18:38:15.060498 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.060456    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:38:15.066243 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.066227    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.066335 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.066269    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:38:15.066335 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.066293    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:38:15.066335 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.066307    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.066454 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.066364    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:38:15.075238 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.075219    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vmn22\""
Oct 30 18:38:15.075690 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.075670    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kp722\""
Oct 30 18:38:15.080825 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.080807    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:38:15.080995 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.080979    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5fxhm\""
Oct 30 18:38:15.081062 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.081003    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kxkbv\""
Oct 30 18:38:15.093413 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093388    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-device-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.093480 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093420    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-etc-selinux\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.093480 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093437    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.093480 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093456    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-socket-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.093480 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093476    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-sys-fs\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.093673 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093573    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxv8\" (UniqueName: \"kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8\") pod \"network-check-target-v7pb2\" (UID: \"de6f539e-a438-42bb-98e6-d83b5926143a\") " pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:38:15.093673 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093608    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.093673 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093657    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.093808 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093698    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf42g\" (UniqueName: \"kubernetes.io/projected/cb8aaa1a-7cea-437a-a436-09a8af560781-kube-api-access-cf42g\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.093808 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093723    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jkqx\" (UniqueName: \"kubernetes.io/projected/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-kube-api-access-7jkqx\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:38:15.093808 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093750    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-serviceca\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:38:15.093808 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093774    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74dqb\" (UniqueName: \"kubernetes.io/projected/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-kube-api-access-74dqb\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:38:15.093808 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093800    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-registration-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.094029 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093822    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffxdt\" (UniqueName: \"kubernetes.io/projected/62eff799-47b9-474a-966f-2689ab4f1c17-kube-api-access-ffxdt\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.094029 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093844    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-system-cni-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.094029 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093865    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.094029 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093905    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:38:15.094029 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.093974    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-host\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:38:15.094029 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.094026    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.094249 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.094033    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-cnibin\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.094249 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.094031    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.094249 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.094116    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.094249 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.094156    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-metrics-certs\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:38:15.094389 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.094280    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb8aaa1a-7cea-437a-a436-09a8af560781-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.094389 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.094381    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-serviceca\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:38:15.101159 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.101140    4853 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:38:15.102112 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.102095    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h66ck\""
Oct 30 18:38:15.104895 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.104874    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:38:15.104959 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.104894    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:38:15.104959 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.104957    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:38:15.105403 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.105389    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p85r8\""
Oct 30 18:38:15.105466 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.105455    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7txc\""
Oct 30 18:38:15.105507 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.105462    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-djndp\""
Oct 30 18:38:15.105507 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.105468    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6qrp7\""
Oct 30 18:38:15.105642 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.105627    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:38:15.105642 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.105637    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:38:15.105715 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.105653    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4k96h\""
Oct 30 18:38:15.105715 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.105666    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:38:15.155743 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.155686    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf42g\" (UniqueName: \"kubernetes.io/projected/cb8aaa1a-7cea-437a-a436-09a8af560781-kube-api-access-cf42g\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.155830 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.155706    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74dqb\" (UniqueName: \"kubernetes.io/projected/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-kube-api-access-74dqb\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:38:15.156172 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.156151    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jkqx\" (UniqueName: \"kubernetes.io/projected/e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1-kube-api-access-7jkqx\") pod \"network-metrics-daemon-w8jdh\" (UID: \"e5c06abd-1f83-4cd9-8eb6-e6efbbbc17c1\") " pod="openshift-multus/network-metrics-daemon-w8jdh"
Oct 30 18:38:15.156465 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.156450    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffxdt\" (UniqueName: \"kubernetes.io/projected/62eff799-47b9-474a-966f-2689ab4f1c17-kube-api-access-ffxdt\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.156522 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.156507    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgxv8\" (UniqueName: \"kubernetes.io/projected/de6f539e-a438-42bb-98e6-d83b5926143a-kube-api-access-sgxv8\") pod \"network-check-target-v7pb2\" (UID: \"de6f539e-a438-42bb-98e6-d83b5926143a\") " pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:38:15.162939 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.162915    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.163024 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.162937    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.163089 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163023    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.163089 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163040    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.163089 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163053    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.163089 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163067    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:38:15.163281 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163074    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.163281 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163133    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:38:15.163281 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163241    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:38:15.163281 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163253    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.163281 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163280    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:38:15.163423 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163283    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.163423 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163290    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:38:15.163423 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163307    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:38:15.163423 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163384    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:38:15.163423 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163413    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:38:15.163580 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.163453    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:38:15.194470 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194453    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-host\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:38:15.194567 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194479    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-dbus\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:15.194567 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194494    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:15.194567 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194510    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-device-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.194567 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194542    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-etc-selinux\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.194759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194572    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e23b1354-07f4-4b59-a6a1-a1c96c8c6b27-host\") pod \"node-ca-q4mnk\" (UID: \"e23b1354-07f4-4b59-a6a1-a1c96c8c6b27\") " pod="openshift-image-registry/node-ca-q4mnk"
Oct 30 18:38:15.194759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194604    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-cert\") pod \"ingress-canary-tkcfx\" (UID: \"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153\") " pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:15.194759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194625    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-etc-selinux\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.194759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194636    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-device-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.194759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194638    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-socket-dir-parent\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.194759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194681    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-slash\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.194945 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194773    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7zhh\" (UniqueName: \"kubernetes.io/projected/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-kube-api-access-j7zhh\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:38:15.194945 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194800    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33db8810-0fe5-4b49-8136-231a566b46e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:15.194945 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194817    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b81698fa-f5d9-48b0-85fa-29e71a85c441-original-pull-secret\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:15.194945 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194844    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-cert\") pod \"ingress-canary-tkcfx\" (UID: \"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153\") " pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:15.194945 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194858    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-k8s-cni-cncf-io\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.194945 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194890    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-multus\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.194945 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194910    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.194945 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194936    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6ls\" (UniqueName: \"kubernetes.io/projected/25811b4f-a869-49f0-ac40-b0f37274e97d-kube-api-access-zk6ls\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194963    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-config\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.194987    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bc74796-475a-4fe8-9113-a6dacb274acd-ovn-node-metrics-cert\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195029    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195056    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/33db8810-0fe5-4b49-8136-231a566b46e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195060    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-netns\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195109    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-metrics-client-ca\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195185    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-config\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195197    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-tmp\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195225    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-netns\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195243    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-script-lib\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195254    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-tmp\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.195257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195257    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47d77dc5-6269-4949-b4fa-60c617073f7c-konnectivity-ca\") pod \"konnectivity-agent-25n6m\" (UID: \"47d77dc5-6269-4949-b4fa-60c617073f7c\") " pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195271    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d48e73c-4c04-452e-976d-cc74ef960a3a-config-volume\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195286    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdnd\" (UniqueName: \"kubernetes.io/projected/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-kube-api-access-lmdnd\") pod \"ingress-canary-tkcfx\" (UID: \"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153\") " pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195307    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-system-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195329    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-conf-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195348    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bc74796-475a-4fe8-9113-a6dacb274acd-ovn-node-metrics-cert\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195353    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwf7\" (UniqueName: \"kubernetes.io/projected/6b567074-99fd-4062-9fbe-a356eb37fe26-kube-api-access-6xwf7\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195392    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-metrics-client-ca\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195392    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-wtmp\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195452    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-ovnkube-script-lib\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195500    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47d77dc5-6269-4949-b4fa-60c617073f7c-konnectivity-ca\") pod \"konnectivity-agent-25n6m\" (UID: \"47d77dc5-6269-4949-b4fa-60c617073f7c\") " pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195500    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-env-overrides\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195566    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhz66\" (UniqueName: \"kubernetes.io/projected/1d48e73c-4c04-452e-976d-cc74ef960a3a-kube-api-access-jhz66\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195594    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-conf\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195615    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d48e73c-4c04-452e-976d-cc74ef960a3a-config-volume\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195652    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bc74796-475a-4fe8-9113-a6dacb274acd-env-overrides\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195693    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-tuned\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195717    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-var-lib-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195739    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195749    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-tuned\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.195796 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195755    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33db8810-0fe5-4b49-8136-231a566b46e5-crio-socket\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195806    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-registration-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195851    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-system-cni-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195873    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-hosts-file\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195904    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-registration-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195909    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-system-cni-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195962    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-daemon-config\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.195991    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196010    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196040    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshdb\" (UniqueName: \"kubernetes.io/projected/8bc74796-475a-4fe8-9113-a6dacb274acd-kube-api-access-lshdb\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196055    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-systemd-units\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196077    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-node-log\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196094    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d48e73c-4c04-452e-976d-cc74ef960a3a-tmp-dir\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196110    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-etc-kubernetes\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196133    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-systemd\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196173    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d48e73c-4c04-452e-976d-cc74ef960a3a-tmp-dir\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196194    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-systemd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196201    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-daemon-config\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196232    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-iptables-alerter-script\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196268    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33db8810-0fe5-4b49-8136-231a566b46e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196353    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d48e73c-4c04-452e-976d-cc74ef960a3a-metrics-tls\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196447    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-iptables-alerter-script\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196478    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196497    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/33db8810-0fe5-4b49-8136-231a566b46e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196506    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-socket-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196519    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196567    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-sys-fs\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196589    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-socket-dir\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196597    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-kubelet-config\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196615    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d48e73c-4c04-452e-976d-cc74ef960a3a-metrics-tls\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196631    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-hostroot\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.196655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196647    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62eff799-47b9-474a-966f-2689ab4f1c17-sys-fs\") pod \"aws-ebs-csi-driver-node-qn8h7\" (UID: \"62eff799-47b9-474a-966f-2689ab4f1c17\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn8h7"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196685    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-sys\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196719    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196743    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-cni-binary-copy\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196760    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-multus-certs\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196780    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-root\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196785    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196823    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196850    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-accelerators-collector-config\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196876    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-kubernetes\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196906    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b567074-99fd-4062-9fbe-a356eb37fe26-cni-binary-copy\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196902    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-cnibin\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196951    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb8aaa1a-7cea-437a-a436-09a8af560781-cnibin\") pod \"multus-additional-cni-plugins-nfgck\" (UID: \"cb8aaa1a-7cea-437a-a436-09a8af560781\") " pod="openshift-multus/multus-additional-cni-plugins-nfgck"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196944    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-bin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.196994    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-lib-modules\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197012    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197028    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-etc-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197035    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-accelerators-collector-config\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197064    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-ovn\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197109    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-bin\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197139    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47d77dc5-6269-4949-b4fa-60c617073f7c-agent-certs\") pod \"konnectivity-agent-25n6m\" (UID: \"47d77dc5-6269-4949-b4fa-60c617073f7c\") " pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197172    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-host-slash\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197187    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-tls\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197204    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lggv5\" (UniqueName: \"kubernetes.io/projected/2bd81706-d26f-420f-a97d-270ebbc96057-kube-api-access-lggv5\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197236    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-run\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197255    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-log-socket\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197270    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-netd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197286    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33db8810-0fe5-4b49-8136-231a566b46e5-data-volume\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197302    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8qw9\" (UniqueName: \"kubernetes.io/projected/33db8810-0fe5-4b49-8136-231a566b46e5-kube-api-access-r8qw9\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197323    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfpk7\" (UniqueName: \"kubernetes.io/projected/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-kube-api-access-dfpk7\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197325    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47d77dc5-6269-4949-b4fa-60c617073f7c-agent-certs\") pod \"konnectivity-agent-25n6m\" (UID: \"47d77dc5-6269-4949-b4fa-60c617073f7c\") " pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197438    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-cnibin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197459    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-tls\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197480    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/33db8810-0fe5-4b49-8136-231a566b46e5-data-volume\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197509    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-textfile\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197562    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-modprobe-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197588    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-sys\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197603    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-textfile\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197612    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-kubelet\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197659    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-tmp-dir\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197688    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-kubelet\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197713    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysconfig\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197722    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-tmp-dir\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197729    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-var-lib-kubelet\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.197839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.197768    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-host\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.260050 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.260015    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfpk7\" (UniqueName: \"kubernetes.io/projected/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-kube-api-access-dfpk7\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:38:15.260235 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.260192    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhz66\" (UniqueName: \"kubernetes.io/projected/1d48e73c-4c04-452e-976d-cc74ef960a3a-kube-api-access-jhz66\") pod \"dns-default-5hwl2\" (UID: \"1d48e73c-4c04-452e-976d-cc74ef960a3a\") " pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:15.260306 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.260267    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdnd\" (UniqueName: \"kubernetes.io/projected/4adb3bc7-4601-4f2c-bce1-52f9ddd3d153-kube-api-access-lmdnd\") pod \"ingress-canary-tkcfx\" (UID: \"4adb3bc7-4601-4f2c-bce1-52f9ddd3d153\") " pod="openshift-ingress-canary/ingress-canary-tkcfx"
Oct 30 18:38:15.260306 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.260275    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7zhh\" (UniqueName: \"kubernetes.io/projected/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-kube-api-access-j7zhh\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:38:15.260417 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.260389    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwf7\" (UniqueName: \"kubernetes.io/projected/6b567074-99fd-4062-9fbe-a356eb37fe26-kube-api-access-6xwf7\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.260811 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.260789    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshdb\" (UniqueName: \"kubernetes.io/projected/8bc74796-475a-4fe8-9113-a6dacb274acd-kube-api-access-lshdb\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.260882 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.260835    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6ls\" (UniqueName: \"kubernetes.io/projected/25811b4f-a869-49f0-ac40-b0f37274e97d-kube-api-access-zk6ls\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.260882 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.260846    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8qw9\" (UniqueName: \"kubernetes.io/projected/33db8810-0fe5-4b49-8136-231a566b46e5-kube-api-access-r8qw9\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:15.260997 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.260982    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lggv5\" (UniqueName: \"kubernetes.io/projected/2bd81706-d26f-420f-a97d-270ebbc96057-kube-api-access-lggv5\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.298310 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298293    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-multus-certs\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.298383 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298316    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-root\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.298383 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298332    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-kubernetes\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.298383 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298348    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-bin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.298383 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298361    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-lib-modules\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298381    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-etc-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298406    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-ovn\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298406    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-root\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298414    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-multus-certs\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298456    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-etc-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298469    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-kubernetes\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298462    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-ovn\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298498    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-bin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298509    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-bin\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298540    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-host-slash\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298554    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-run\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298558    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-bin\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298561    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-lib-modules\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298568    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-log-socket\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298589    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-run\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298598    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-netd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298606    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-log-socket\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.298611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298586    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed3bc506-2acc-4de7-8d28-738da0eaa6ee-host-slash\") pod \"iptables-alerter-hhdgg\" (UID: \"ed3bc506-2acc-4de7-8d28-738da0eaa6ee\") " pod="openshift-network-operator/iptables-alerter-hhdgg"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298630    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-cni-netd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298640    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-cnibin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298664    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-modprobe-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298721    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-modprobe-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298753    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-cnibin\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298754    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-sys\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298796    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-sys\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298820    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-kubelet\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298843    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-kubelet\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298878    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-kubelet\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298892    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysconfig\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298906    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-var-lib-kubelet\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298921    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-host\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298937    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-dbus\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298953    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-socket-dir-parent\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298966    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-slash\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298972    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysconfig\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298972    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-var-lib-kubelet\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298983    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-k8s-cni-cncf-io\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.298996    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-host\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299014    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-kubelet\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299024    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-slash\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299025    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-multus\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299036    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-socket-dir-parent\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299042    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-k8s-cni-cncf-io\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299037    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-dbus\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299056    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299096    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-var-lib-cni-multus\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299153    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299178    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-netns\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299227    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-d\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.299225 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299242    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-netns\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299254    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299261    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-host-run-netns\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299294    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-netns\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299264    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-system-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299310    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-system-cni-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299327    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-conf-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299345    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-wtmp\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299366    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-conf\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299371    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-multus-conf-dir\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299383    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-var-lib-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299406    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299427    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33db8810-0fe5-4b49-8136-231a566b46e5-crio-socket\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299447    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-hosts-file\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299448    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-var-lib-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299452    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299470    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299474    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-node-exporter-wtmp\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299490    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/33db8810-0fe5-4b49-8136-231a566b46e5-crio-socket\") pod \"insights-runtime-extractor-bjg6d\" (UID: \"33db8810-0fe5-4b49-8136-231a566b46e5\") " pod="openshift-insights/insights-runtime-extractor-bjg6d"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299497    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-sysctl-conf\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299520    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299555    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-openvswitch\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299570    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-systemd-units\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299570    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-host-run-ovn-kubernetes\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299591    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-node-log\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299605    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-etc-kubernetes\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299525    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c-hosts-file\") pod \"node-resolver-tnkkl\" (UID: \"c0c2b89c-c9e8-4bc7-a733-e9ba76cde59c\") " pod="openshift-dns/node-resolver-tnkkl"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299618    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-systemd\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299641    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-systemd-units\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299659    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-node-log\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299671    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-etc-kubernetes\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299666    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25811b4f-a869-49f0-ac40-b0f37274e97d-etc-systemd\") pod \"tuned-ckgwd\" (UID: \"25811b4f-a869-49f0-ac40-b0f37274e97d\") " pod="openshift-cluster-node-tuning-operator/tuned-ckgwd"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299684    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-systemd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299698    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bc74796-475a-4fe8-9113-a6dacb274acd-run-systemd\") pod \"ovnkube-node-7k5m2\" (UID: \"8bc74796-475a-4fe8-9113-a6dacb274acd\") " pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299715    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-kubelet-config\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299738    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-hostroot\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299750    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-sys\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299806    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd81706-d26f-420f-a97d-270ebbc96057-sys\") pod \"node-exporter-bjkm5\" (UID: \"2bd81706-d26f-420f-a97d-270ebbc96057\") " pod="openshift-monitoring/node-exporter-bjkm5"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299811    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b567074-99fd-4062-9fbe-a356eb37fe26-hostroot\") pod \"multus-26flh\" (UID: \"6b567074-99fd-4062-9fbe-a356eb37fe26\") " pod="openshift-multus/multus-26flh"
Oct 30 18:38:15.300296 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.299808    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b81698fa-f5d9-48b0-85fa-29e71a85c441-kubelet-config\") pod \"global-pull-secret-syncer-mqdgz\" (UID: \"b81698fa-f5d9-48b0-85fa-29e71a85c441\") " pod="kube-system/global-pull-secret-syncer-mqdgz"
Oct 30 18:38:15.538594 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.538500    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:15.974121 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:15.974082    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:16.988880 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:16.988849    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:38:18.123868 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:18.123830    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:18.125714 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:18.125690    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5hwl2"
Oct 30 18:38:19.874853 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:19.874814    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:38:22.719037 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:22.719003    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:24.175515 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:24.175479    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:24.175990 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:24.175695    4853 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:38:24.694502 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:24.694480    4853 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-31.us-west-2.compute.internal" event="NodeReady"
Oct 30 18:38:25.411979 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:25.411948    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:25.794707 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:25.794623    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:26.179855 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:26.179821    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:31.266041 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:31.266006    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:31.676562 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:31.676505    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:39.371146 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:39.371110    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7k5m2"
Oct 30 18:38:39.764567 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:39.764475    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:39.971942 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:39.971912    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:44.103057 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:44.103020    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-25n6m"
Oct 30 18:38:46.992194 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:46.992160    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v7pb2"
Oct 30 18:38:48.501602 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:48.501562    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:49.873200 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:38:49.873164    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:38:51.939633 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:51.939593    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:52.197157 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:38:52.197079    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:19.872324 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:39:19.872288    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:39:33.197323 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:39:33.197280    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:33.393263 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:39:33.393192    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:42.672308 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:39:42.672264    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:42.879470 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:39:42.879432    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:43.086989 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:39:43.086955    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:43.286227 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:39:43.286200    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:45.880673 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:39:45.880633    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:46.078056 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:39:46.078017    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:48.833442 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:39:48.833404    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:49.874737 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:39:49.874699    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:39:53.665037 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:39:53.664997    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:53.877523 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:39:53.877493    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:19.874307 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:40:19.874266    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:40:31.897940 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:40:31.897887    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:32.103751 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:40:32.103726    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:49.165749 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:40:49.165707    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:49.429723 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:40:49.429641    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:49.874973 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:40:49.874936    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:41:19.874970 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:41:19.874918    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:41:26.327286 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:41:26.327248    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:49.521658 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:41:49.521618    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:49.875458 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:41:49.875426    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:42:05.655274 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:05.655234    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:05.857112 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:05.857076    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:14.067881 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:14.067825    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:15.444086 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:15.444045    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:19.875323 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:42:19.875281    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:42:30.439591 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:30.439551    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:30.645583 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:30.645553    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:31.459231 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.459184    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"]
Oct 30 18:42:31.461345 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.461325    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:31.552783 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.552755    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:31.552944 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.552797    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:31.552944 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.552819    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q924m\" (UniqueName: \"kubernetes.io/projected/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-kube-api-access-q924m\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:31.557664 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.557645    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"98a21a31f7a8ae0212b9c22cc643f9202bfa37-6b211c7f-dockercfg-45pdq\""
Oct 30 18:42:31.557725 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.557644    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:42:31.557725 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.557695    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:42:31.558539 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.558508    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"]
Oct 30 18:42:31.653603 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.653568    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:31.653715 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.653612    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:31.653715 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.653638    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q924m\" (UniqueName: \"kubernetes.io/projected/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-kube-api-access-q924m\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:31.653931 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.653904    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:31.653990 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.653974    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:31.706044 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.706020    4853 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:42:31.708384 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.708365    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q924m\" (UniqueName: \"kubernetes.io/projected/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-kube-api-access-q924m\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:31.774326 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:31.774280    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:32.138210 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:32.138183    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"]
Oct 30 18:42:32.145881 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:42:32.145854    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf786b4b3_8047_4988_a6b4_7c0e5626ef2f.slice/crio-84b8c281b9f1262b41436b4ba9776d329da9e0e730a0dcb5c2e226f8f8e9945c WatchSource:0}: Error finding container 84b8c281b9f1262b41436b4ba9776d329da9e0e730a0dcb5c2e226f8f8e9945c: Status 404 returned error can't find the container with id 84b8c281b9f1262b41436b4ba9776d329da9e0e730a0dcb5c2e226f8f8e9945c
Oct 30 18:42:32.147852 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:32.147833    4853 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:42:32.770260 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:32.770230    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk" event={"ID":"f786b4b3-8047-4988-a6b4-7c0e5626ef2f","Type":"ContainerStarted","Data":"84b8c281b9f1262b41436b4ba9776d329da9e0e730a0dcb5c2e226f8f8e9945c"}
Oct 30 18:42:38.790522 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:38.790486    4853 generic.go:356] "Generic (PLEG): container finished" podID="f786b4b3-8047-4988-a6b4-7c0e5626ef2f" containerID="ecca8537888ebee0dc7ce0906ab357af9172595dfe6ac4dfc3632b48685c9909" exitCode=0
Oct 30 18:42:38.791017 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:38.790572    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk" event={"ID":"f786b4b3-8047-4988-a6b4-7c0e5626ef2f","Type":"ContainerDied","Data":"ecca8537888ebee0dc7ce0906ab357af9172595dfe6ac4dfc3632b48685c9909"}
Oct 30 18:42:42.803434 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:42.803402    4853 generic.go:356] "Generic (PLEG): container finished" podID="f786b4b3-8047-4988-a6b4-7c0e5626ef2f" containerID="2ebd1197e655a2640c53d29cff8088f081fd214c4d6025c9be0efe0864ca11fa" exitCode=0
Oct 30 18:42:42.803803 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:42.803454    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk" event={"ID":"f786b4b3-8047-4988-a6b4-7c0e5626ef2f","Type":"ContainerDied","Data":"2ebd1197e655a2640c53d29cff8088f081fd214c4d6025c9be0efe0864ca11fa"}
Oct 30 18:42:47.571115 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:47.571076    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:47.781397 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:47.781364    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:49.827373 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:49.827329    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk" event={"ID":"f786b4b3-8047-4988-a6b4-7c0e5626ef2f","Type":"ContainerStarted","Data":"360bafffbceaaea1094119f51ccc50d8c2b583516732bd4f3cf6f91d79aff8e9"}
Oct 30 18:42:49.874472 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:42:49.874439    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:42:49.914486 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:49.914465    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:50.833378 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:50.833344    4853 generic.go:356] "Generic (PLEG): container finished" podID="f786b4b3-8047-4988-a6b4-7c0e5626ef2f" containerID="360bafffbceaaea1094119f51ccc50d8c2b583516732bd4f3cf6f91d79aff8e9" exitCode=0
Oct 30 18:42:50.833779 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:50.833443    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk" event={"ID":"f786b4b3-8047-4988-a6b4-7c0e5626ef2f","Type":"ContainerDied","Data":"360bafffbceaaea1094119f51ccc50d8c2b583516732bd4f3cf6f91d79aff8e9"}
Oct 30 18:42:51.959275 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:51.959256    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:52.091734 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.091684    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-util\" (UniqueName: \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-util\") pod \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") "
Oct 30 18:42:52.091734 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.091726    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-kube-api-access-q924m\" (UniqueName: \"kubernetes.io/projected/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-kube-api-access-q924m\") pod \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") "
Oct 30 18:42:52.091875 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.091761    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-bundle\") pod \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\" (UID: \"f786b4b3-8047-4988-a6b4-7c0e5626ef2f\") "
Oct 30 18:42:52.092648 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.092611    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-bundle" pod "f786b4b3-8047-4988-a6b4-7c0e5626ef2f" (UID: "f786b4b3-8047-4988-a6b4-7c0e5626ef2f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:52.093851 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.093826    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-kube-api-access-q924m" pod "f786b4b3-8047-4988-a6b4-7c0e5626ef2f" (UID: "f786b4b3-8047-4988-a6b4-7c0e5626ef2f"). InnerVolumeSpecName "kube-api-access-q924m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:52.096760 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.096740    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-util" pod "f786b4b3-8047-4988-a6b4-7c0e5626ef2f" (UID: "f786b4b3-8047-4988-a6b4-7c0e5626ef2f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:52.193015 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.192992    4853 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-util\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:52.193015 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.193013    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q924m\" (UniqueName: \"kubernetes.io/projected/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-kube-api-access-q924m\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:52.193126 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.193022    4853 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f786b4b3-8047-4988-a6b4-7c0e5626ef2f-bundle\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:52.844933 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.844902    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk" event={"ID":"f786b4b3-8047-4988-a6b4-7c0e5626ef2f","Type":"ContainerDied","Data":"84b8c281b9f1262b41436b4ba9776d329da9e0e730a0dcb5c2e226f8f8e9945c"}
Oct 30 18:42:52.845099 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.844947    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cl7cdk"
Oct 30 18:42:52.845099 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:52.844955    4853 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84b8c281b9f1262b41436b4ba9776d329da9e0e730a0dcb5c2e226f8f8e9945c"
Oct 30 18:42:57.841882 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:57.841842    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:58.058706 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:58.058671    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:58.979686 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:58.979649    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"]
Oct 30 18:42:58.980101 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:58.979869    4853 cpu_manager.go:414] "containerMap: removing container" podUID="f786b4b3-8047-4988-a6b4-7c0e5626ef2f" containerName="util"
Oct 30 18:42:58.980101 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:58.979880    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f786b4b3-8047-4988-a6b4-7c0e5626ef2f" containerName="util"
Oct 30 18:42:58.980101 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:58.979891    4853 cpu_manager.go:414] "containerMap: removing container" podUID="f786b4b3-8047-4988-a6b4-7c0e5626ef2f" containerName="extract"
Oct 30 18:42:58.980101 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:58.979896    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f786b4b3-8047-4988-a6b4-7c0e5626ef2f" containerName="extract"
Oct 30 18:42:58.980101 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:58.979915    4853 cpu_manager.go:414] "containerMap: removing container" podUID="f786b4b3-8047-4988-a6b4-7c0e5626ef2f" containerName="pull"
Oct 30 18:42:58.980101 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:58.979920    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f786b4b3-8047-4988-a6b4-7c0e5626ef2f" containerName="pull"
Oct 30 18:42:58.980101 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:58.979964    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="f786b4b3-8047-4988-a6b4-7c0e5626ef2f" containerName="extract"
Oct 30 18:42:58.981810 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:58.981791    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"
Oct 30 18:42:59.034457 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.034431    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26m2b\" (UniqueName: \"kubernetes.io/projected/433a9945-c903-4a40-a406-c26d140d9e48-kube-api-access-26m2b\") pod \"custom-metrics-autoscaler-operator-c667576c8-kx9cd\" (UID: \"433a9945-c903-4a40-a406-c26d140d9e48\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"
Oct 30 18:42:59.034617 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.034466    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/433a9945-c903-4a40-a406-c26d140d9e48-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-kx9cd\" (UID: \"433a9945-c903-4a40-a406-c26d140d9e48\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"
Oct 30 18:42:59.043860 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.043843    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Oct 30 18:42:59.043926 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.043866    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Oct 30 18:42:59.089369 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.089347    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Oct 30 18:42:59.089523 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.089505    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-9fzqg\""
Oct 30 18:42:59.090132 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.090115    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"]
Oct 30 18:42:59.135439 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.135416    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26m2b\" (UniqueName: \"kubernetes.io/projected/433a9945-c903-4a40-a406-c26d140d9e48-kube-api-access-26m2b\") pod \"custom-metrics-autoscaler-operator-c667576c8-kx9cd\" (UID: \"433a9945-c903-4a40-a406-c26d140d9e48\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"
Oct 30 18:42:59.135603 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.135457    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/433a9945-c903-4a40-a406-c26d140d9e48-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-kx9cd\" (UID: \"433a9945-c903-4a40-a406-c26d140d9e48\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"
Oct 30 18:42:59.137910 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.137889    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/433a9945-c903-4a40-a406-c26d140d9e48-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-kx9cd\" (UID: \"433a9945-c903-4a40-a406-c26d140d9e48\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"
Oct 30 18:42:59.191478 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.191449    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26m2b\" (UniqueName: \"kubernetes.io/projected/433a9945-c903-4a40-a406-c26d140d9e48-kube-api-access-26m2b\") pod \"custom-metrics-autoscaler-operator-c667576c8-kx9cd\" (UID: \"433a9945-c903-4a40-a406-c26d140d9e48\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"
Oct 30 18:42:59.293390 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.293335    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"
Oct 30 18:42:59.726034 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.726003    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"]
Oct 30 18:42:59.733638 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:42:59.733610    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433a9945_c903_4a40_a406_c26d140d9e48.slice/crio-515c24fd359dec8be3760f58d2d4324a6006a244b565c870e6cc4846e3dc6449 WatchSource:0}: Error finding container 515c24fd359dec8be3760f58d2d4324a6006a244b565c870e6cc4846e3dc6449: Status 404 returned error can't find the container with id 515c24fd359dec8be3760f58d2d4324a6006a244b565c870e6cc4846e3dc6449
Oct 30 18:42:59.868355 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:42:59.868328    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd" event={"ID":"433a9945-c903-4a40-a406-c26d140d9e48","Type":"ContainerStarted","Data":"515c24fd359dec8be3760f58d2d4324a6006a244b565c870e6cc4846e3dc6449"}
Oct 30 18:43:03.882035 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:03.881999    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd" event={"ID":"433a9945-c903-4a40-a406-c26d140d9e48","Type":"ContainerStarted","Data":"d1f15a4c440d5c4f9a4037d9cded4703d138617ce6b6ebb12378d4384237c987"}
Oct 30 18:43:03.882402 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:03.882137    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"
Oct 30 18:43:04.086933 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:04.086897    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd" podStartSLOduration=2.060171735 podStartE2EDuration="6.086885155s" totalImagesPullingTime="4.02671342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:04.040408092 +0000 UTC m=+290.210764940" watchObservedRunningTime="2026-10-30 18:43:04.086885155 +0000 UTC m=+290.257242002"
Oct 30 18:43:04.938076 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:04.938036    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:06.630639 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.630604    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-7dc85965f-tskt9"]
Oct 30 18:43:06.633937 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.633914    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:06.635336 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.635306    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:06.687843 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.687818    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-7dc85965f-tskt9"]
Oct 30 18:43:06.730623 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.730601    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Oct 30 18:43:06.730623 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.730607    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Oct 30 18:43:06.730769 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.730662    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-lxl88\""
Oct 30 18:43:06.783668 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.783639    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjb5d\" (UniqueName: \"kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-kube-api-access-bjb5d\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:06.783775 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.783687    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-cabundle0\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:06.783849 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.783812    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-certificates\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:06.884696 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.884647    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-certificates\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:06.884696 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.884689    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjb5d\" (UniqueName: \"kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-kube-api-access-bjb5d\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:06.884846 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.884710    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-cabundle0\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:06.884846 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:06.884778    4853 configmap.go:193] Couldn't get configMap openshift-keda/keda-ocp-cabundle: configmap "keda-ocp-cabundle" not found
Oct 30 18:43:06.884846 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:06.884819    4853 secret.go:281] references non-existent secret key: ca.crt
Oct 30 18:43:06.884846 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:06.884839    4853 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-cabundle0 podName:d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9 nodeName:}" failed. No retries permitted until 2026-10-30 18:43:07.38482438 +0000 UTC m=+293.555181221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cabundle0" (UniqueName: "kubernetes.io/configmap/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-cabundle0") pod "keda-operator-7dc85965f-tskt9" (UID: "d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9") : configmap "keda-ocp-cabundle" not found
Oct 30 18:43:06.884846 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:06.884839    4853 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Oct 30 18:43:06.885046 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:06.884853    4853 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-7dc85965f-tskt9: references non-existent secret key: ca.crt
Oct 30 18:43:06.885046 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:06.884908    4853 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-certificates podName:d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9 nodeName:}" failed. No retries permitted until 2026-10-30 18:43:07.384890655 +0000 UTC m=+293.555247496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-certificates") pod "keda-operator-7dc85965f-tskt9" (UID: "d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9") : references non-existent secret key: ca.crt
Oct 30 18:43:06.900348 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.900327    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:06.940012 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:06.939993    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjb5d\" (UniqueName: \"kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-kube-api-access-bjb5d\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:07.387581 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:07.387562    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-cabundle0\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:07.387708 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:07.387595    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-certificates\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:07.387708 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:07.387673    4853 secret.go:281] references non-existent secret key: ca.crt
Oct 30 18:43:07.387708 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:07.387684    4853 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Oct 30 18:43:07.387708 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:07.387692    4853 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-7dc85965f-tskt9: references non-existent secret key: ca.crt
Oct 30 18:43:07.387842 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:07.387744    4853 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-certificates podName:d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9 nodeName:}" failed. No retries permitted until 2026-10-30 18:43:08.387731146 +0000 UTC m=+294.558087972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-certificates") pod "keda-operator-7dc85965f-tskt9" (UID: "d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9") : references non-existent secret key: ca.crt
Oct 30 18:43:07.388165 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:07.388148    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-cabundle0\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:07.876367 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:07.876331    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:07.895563 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:07.895521    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"]
Oct 30 18:43:07.898902 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:07.898883    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:07.952214 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:07.952184    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"]
Oct 30 18:43:07.960798 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:07.960781    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Oct 30 18:43:07.992983 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:07.992962    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8ab46ba4-1ad5-42b0-85e8-57eff60a87ab-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-h9cfm\" (UID: \"8ab46ba4-1ad5-42b0-85e8-57eff60a87ab\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:07.993062 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:07.993019    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ab46ba4-1ad5-42b0-85e8-57eff60a87ab-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-h9cfm\" (UID: \"8ab46ba4-1ad5-42b0-85e8-57eff60a87ab\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:07.993104 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:07.993088    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsl77\" (UniqueName: \"kubernetes.io/projected/8ab46ba4-1ad5-42b0-85e8-57eff60a87ab-kube-api-access-vsl77\") pod \"keda-metrics-apiserver-86bd856cfb-h9cfm\" (UID: \"8ab46ba4-1ad5-42b0-85e8-57eff60a87ab\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:08.076972 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.076948    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:08.093964 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.093941    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ab46ba4-1ad5-42b0-85e8-57eff60a87ab-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-h9cfm\" (UID: \"8ab46ba4-1ad5-42b0-85e8-57eff60a87ab\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:08.094026 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.093972    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsl77\" (UniqueName: \"kubernetes.io/projected/8ab46ba4-1ad5-42b0-85e8-57eff60a87ab-kube-api-access-vsl77\") pod \"keda-metrics-apiserver-86bd856cfb-h9cfm\" (UID: \"8ab46ba4-1ad5-42b0-85e8-57eff60a87ab\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:08.094026 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.093994    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8ab46ba4-1ad5-42b0-85e8-57eff60a87ab-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-h9cfm\" (UID: \"8ab46ba4-1ad5-42b0-85e8-57eff60a87ab\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:08.094287 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.094275    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8ab46ba4-1ad5-42b0-85e8-57eff60a87ab-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-h9cfm\" (UID: \"8ab46ba4-1ad5-42b0-85e8-57eff60a87ab\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:08.096563 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.096548    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ab46ba4-1ad5-42b0-85e8-57eff60a87ab-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-h9cfm\" (UID: \"8ab46ba4-1ad5-42b0-85e8-57eff60a87ab\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:08.148977 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.148928    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsl77\" (UniqueName: \"kubernetes.io/projected/8ab46ba4-1ad5-42b0-85e8-57eff60a87ab-kube-api-access-vsl77\") pod \"keda-metrics-apiserver-86bd856cfb-h9cfm\" (UID: \"8ab46ba4-1ad5-42b0-85e8-57eff60a87ab\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:08.210948 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.210927    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:08.352604 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.352576    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:08.369066 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.369045    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-g8rcq"]
Oct 30 18:43:08.372079 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.372063    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-g8rcq"
Oct 30 18:43:08.396580 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.396559    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-certificates\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:08.398860 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.398840    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9-certificates\") pod \"keda-operator-7dc85965f-tskt9\" (UID: \"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9\") " pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:08.422675 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.422632    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Oct 30 18:43:08.428755 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.428731    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-g8rcq"]
Oct 30 18:43:08.448118 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.448098    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:08.497232 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.497047    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhdp\" (UniqueName: \"kubernetes.io/projected/1b52011f-089f-4e51-af70-ffe2b315bd3f-kube-api-access-pmhdp\") pod \"keda-admission-7b7c985c58-g8rcq\" (UID: \"1b52011f-089f-4e51-af70-ffe2b315bd3f\") " pod="openshift-keda/keda-admission-7b7c985c58-g8rcq"
Oct 30 18:43:08.497232 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.497142    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1b52011f-089f-4e51-af70-ffe2b315bd3f-certificates\") pod \"keda-admission-7b7c985c58-g8rcq\" (UID: \"1b52011f-089f-4e51-af70-ffe2b315bd3f\") " pod="openshift-keda/keda-admission-7b7c985c58-g8rcq"
Oct 30 18:43:08.562427 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.562403    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:08.577015 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.576992    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"]
Oct 30 18:43:08.584356 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:43:08.584332    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab46ba4_1ad5_42b0_85e8_57eff60a87ab.slice/crio-77e1ab978f7b07f862b1926cbfd69cd738652de659902875e563b6c1407609e0 WatchSource:0}: Error finding container 77e1ab978f7b07f862b1926cbfd69cd738652de659902875e563b6c1407609e0: Status 404 returned error can't find the container with id 77e1ab978f7b07f862b1926cbfd69cd738652de659902875e563b6c1407609e0
Oct 30 18:43:08.598397 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.598375    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhdp\" (UniqueName: \"kubernetes.io/projected/1b52011f-089f-4e51-af70-ffe2b315bd3f-kube-api-access-pmhdp\") pod \"keda-admission-7b7c985c58-g8rcq\" (UID: \"1b52011f-089f-4e51-af70-ffe2b315bd3f\") " pod="openshift-keda/keda-admission-7b7c985c58-g8rcq"
Oct 30 18:43:08.598501 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.598420    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1b52011f-089f-4e51-af70-ffe2b315bd3f-certificates\") pod \"keda-admission-7b7c985c58-g8rcq\" (UID: \"1b52011f-089f-4e51-af70-ffe2b315bd3f\") " pod="openshift-keda/keda-admission-7b7c985c58-g8rcq"
Oct 30 18:43:08.600843 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.600822    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1b52011f-089f-4e51-af70-ffe2b315bd3f-certificates\") pod \"keda-admission-7b7c985c58-g8rcq\" (UID: \"1b52011f-089f-4e51-af70-ffe2b315bd3f\") " pod="openshift-keda/keda-admission-7b7c985c58-g8rcq"
Oct 30 18:43:08.653886 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.653866    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhdp\" (UniqueName: \"kubernetes.io/projected/1b52011f-089f-4e51-af70-ffe2b315bd3f-kube-api-access-pmhdp\") pod \"keda-admission-7b7c985c58-g8rcq\" (UID: \"1b52011f-089f-4e51-af70-ffe2b315bd3f\") " pod="openshift-keda/keda-admission-7b7c985c58-g8rcq"
Oct 30 18:43:08.684445 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.684395    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-g8rcq"
Oct 30 18:43:08.761309 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.761266    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:08.814258 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.814235    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-7dc85965f-tskt9"]
Oct 30 18:43:08.821050 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:43:08.821027    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e0b4ea_f13d_4ec1_8390_1b94bfd3f0f9.slice/crio-466c112134045253a6f271409a36b08746dc2e41c8c118bdacfb38564425dc7d WatchSource:0}: Error finding container 466c112134045253a6f271409a36b08746dc2e41c8c118bdacfb38564425dc7d: Status 404 returned error can't find the container with id 466c112134045253a6f271409a36b08746dc2e41c8c118bdacfb38564425dc7d
Oct 30 18:43:08.900504 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.900474    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-7dc85965f-tskt9" event={"ID":"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9","Type":"ContainerStarted","Data":"466c112134045253a6f271409a36b08746dc2e41c8c118bdacfb38564425dc7d"}
Oct 30 18:43:08.901667 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:08.901646    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm" event={"ID":"8ab46ba4-1ad5-42b0-85e8-57eff60a87ab","Type":"ContainerStarted","Data":"77e1ab978f7b07f862b1926cbfd69cd738652de659902875e563b6c1407609e0"}
Oct 30 18:43:09.049885 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:09.049824    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-g8rcq"]
Oct 30 18:43:09.057108 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:43:09.057080    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b52011f_089f_4e51_af70_ffe2b315bd3f.slice/crio-8f609d3323180c555739434b95ae4903f29b4d2b5cc633e791efd041c6655b7a WatchSource:0}: Error finding container 8f609d3323180c555739434b95ae4903f29b4d2b5cc633e791efd041c6655b7a: Status 404 returned error can't find the container with id 8f609d3323180c555739434b95ae4903f29b4d2b5cc633e791efd041c6655b7a
Oct 30 18:43:09.906173 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:09.906134    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-g8rcq" event={"ID":"1b52011f-089f-4e51-af70-ffe2b315bd3f","Type":"ContainerStarted","Data":"8f609d3323180c555739434b95ae4903f29b4d2b5cc633e791efd041c6655b7a"}
Oct 30 18:43:10.706277 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:10.706221    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:11.650452 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:11.650405    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:12.142304 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:12.142259    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:12.365178 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:12.365091    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:12.921465 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:12.921420    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-g8rcq" event={"ID":"1b52011f-089f-4e51-af70-ffe2b315bd3f","Type":"ContainerStarted","Data":"ed89eb28e3e6b4332bc8f99559d95f9e0760c54e29a4d012ca4dd2f5091639ff"}
Oct 30 18:43:12.921956 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:12.921580    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-7b7c985c58-g8rcq"
Oct 30 18:43:13.048697 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:13.048633    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-admission-7b7c985c58-g8rcq" podStartSLOduration=2.177851154 podStartE2EDuration="5.048614215s" totalImagesPullingTime="2.870763061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:13.047520567 +0000 UTC m=+299.217877415" watchObservedRunningTime="2026-10-30 18:43:13.048614215 +0000 UTC m=+299.218971065"
Oct 30 18:43:13.940087 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:13.939975    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:13.940087 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:13.940037    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm" event={"ID":"8ab46ba4-1ad5-42b0-85e8-57eff60a87ab","Type":"ContainerStarted","Data":"d7855656dfaafcfe6ecab3f8caf9d9bb6ab946a09f1a3571541b3ca8637a1b5d"}
Oct 30 18:43:14.047839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:14.047792    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm" podStartSLOduration=2.398802843 podStartE2EDuration="7.047773094s" totalImagesPullingTime="4.648970251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:14.047215759 +0000 UTC m=+300.217572611" watchObservedRunningTime="2026-10-30 18:43:14.047773094 +0000 UTC m=+300.218129942"
Oct 30 18:43:14.941209 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:14.941175    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-7dc85965f-tskt9" event={"ID":"d5e0b4ea-f13d-4ec1-8390-1b94bfd3f0f9","Type":"ContainerStarted","Data":"d3ee00784bd26f81c9b082209e021a82c40a938d8d9734a3ca002e7be0acecfd"}
Oct 30 18:43:14.941635 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:14.941279    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:15.051276 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:15.051234    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-operator-7dc85965f-tskt9" podStartSLOduration=3.677706755 podStartE2EDuration="9.051216786s" totalImagesPullingTime="5.373510031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:15.050507982 +0000 UTC m=+301.220864832" watchObservedRunningTime="2026-10-30 18:43:15.051216786 +0000 UTC m=+301.221573635"
Oct 30 18:43:19.873247 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:19.873207    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:43:24.890019 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:24.889985    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-kx9cd"
Oct 30 18:43:24.946617 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:24.946590    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-h9cfm"
Oct 30 18:43:33.938550 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:33.938506    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-7b7c985c58-g8rcq"
Oct 30 18:43:35.947348 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:35.947323    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-7dc85965f-tskt9"
Oct 30 18:43:49.876080 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:43:49.876036    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:43:50.323651 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:50.323569    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:50.550990 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:50.550938    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:58.258632 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.258594    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:58.450214 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.450180    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"]
Oct 30 18:43:58.452387 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.452367    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:43:58.521025 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.520964    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:58.548968 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.548947    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:43:58.549283 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.549263    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:43:58.549497 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.549484    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-5h2rz\""
Oct 30 18:43:58.550229 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.550211    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"]
Oct 30 18:43:58.621181 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.621155    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:43:58.621308 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.621204    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97bj\" (UniqueName: \"kubernetes.io/projected/8b0ef6db-1c00-4359-bb19-6390bc20b64f-kube-api-access-m97bj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:43:58.621308 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.621281    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:43:58.721999 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.721976    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:43:58.722102 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.722022    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97bj\" (UniqueName: \"kubernetes.io/projected/8b0ef6db-1c00-4359-bb19-6390bc20b64f-kube-api-access-m97bj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:43:58.722102 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.722044    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:43:58.722372 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.722352    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:43:58.722442 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.722424    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:43:58.781715 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:58.781654    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97bj\" (UniqueName: \"kubernetes.io/projected/8b0ef6db-1c00-4359-bb19-6390bc20b64f-kube-api-access-m97bj\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:43:59.066375 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:59.066323    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:43:59.439310 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:43:59.439268    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"]
Oct 30 18:43:59.472751 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:43:59.472720    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0ef6db_1c00_4359_bb19_6390bc20b64f.slice/crio-1bc5f8eca348b1e060f753e65fe8f912de835485539036ce667260838fb4d3a7 WatchSource:0}: Error finding container 1bc5f8eca348b1e060f753e65fe8f912de835485539036ce667260838fb4d3a7: Status 404 returned error can't find the container with id 1bc5f8eca348b1e060f753e65fe8f912de835485539036ce667260838fb4d3a7
Oct 30 18:44:00.084832 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:00.084797    4853 generic.go:356] "Generic (PLEG): container finished" podID="8b0ef6db-1c00-4359-bb19-6390bc20b64f" containerID="24a891c6a6eccc5ca8ef2c0b30cffd48e754d3f59147837bc3429e6f5e20cfb0" exitCode=0
Oct 30 18:44:00.084970 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:00.084883    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g" event={"ID":"8b0ef6db-1c00-4359-bb19-6390bc20b64f","Type":"ContainerDied","Data":"24a891c6a6eccc5ca8ef2c0b30cffd48e754d3f59147837bc3429e6f5e20cfb0"}
Oct 30 18:44:00.084970 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:00.084937    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g" event={"ID":"8b0ef6db-1c00-4359-bb19-6390bc20b64f","Type":"ContainerStarted","Data":"1bc5f8eca348b1e060f753e65fe8f912de835485539036ce667260838fb4d3a7"}
Oct 30 18:44:03.096747 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:03.096714    4853 generic.go:356] "Generic (PLEG): container finished" podID="8b0ef6db-1c00-4359-bb19-6390bc20b64f" containerID="d34e27a699df35f3986fd29755a98e12fffa6341222fb865a83c347475487be4" exitCode=0
Oct 30 18:44:03.097085 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:03.096810    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g" event={"ID":"8b0ef6db-1c00-4359-bb19-6390bc20b64f","Type":"ContainerDied","Data":"d34e27a699df35f3986fd29755a98e12fffa6341222fb865a83c347475487be4"}
Oct 30 18:44:04.101676 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:04.101645    4853 generic.go:356] "Generic (PLEG): container finished" podID="8b0ef6db-1c00-4359-bb19-6390bc20b64f" containerID="7c8d56bd1a614e750d30085a979a3d2b059aa71654d8bb37164e1ef6858bdb34" exitCode=0
Oct 30 18:44:04.102028 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:04.101732    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g" event={"ID":"8b0ef6db-1c00-4359-bb19-6390bc20b64f","Type":"ContainerDied","Data":"7c8d56bd1a614e750d30085a979a3d2b059aa71654d8bb37164e1ef6858bdb34"}
Oct 30 18:44:05.219135 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:05.219111    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:44:05.367218 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:05.367158    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0ef6db-1c00-4359-bb19-6390bc20b64f-kube-api-access-m97bj\" (UniqueName: \"kubernetes.io/projected/8b0ef6db-1c00-4359-bb19-6390bc20b64f-kube-api-access-m97bj\") pod \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") "
Oct 30 18:44:05.367218 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:05.367205    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-bundle\") pod \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") "
Oct 30 18:44:05.367356 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:05.367310    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-util\" (UniqueName: \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-util\") pod \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\" (UID: \"8b0ef6db-1c00-4359-bb19-6390bc20b64f\") "
Oct 30 18:44:05.367885 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:05.367862    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-bundle" pod "8b0ef6db-1c00-4359-bb19-6390bc20b64f" (UID: "8b0ef6db-1c00-4359-bb19-6390bc20b64f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:05.369311 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:05.369290    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0ef6db-1c00-4359-bb19-6390bc20b64f-kube-api-access-m97bj" pod "8b0ef6db-1c00-4359-bb19-6390bc20b64f" (UID: "8b0ef6db-1c00-4359-bb19-6390bc20b64f"). InnerVolumeSpecName "kube-api-access-m97bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:44:05.371716 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:05.371696    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-util" pod "8b0ef6db-1c00-4359-bb19-6390bc20b64f" (UID: "8b0ef6db-1c00-4359-bb19-6390bc20b64f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:05.468331 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:05.468299    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m97bj\" (UniqueName: \"kubernetes.io/projected/8b0ef6db-1c00-4359-bb19-6390bc20b64f-kube-api-access-m97bj\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:05.468331 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:05.468329    4853 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-bundle\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:05.468331 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:05.468339    4853 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0ef6db-1c00-4359-bb19-6390bc20b64f-util\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:06.110669 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:06.110644    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g"
Oct 30 18:44:06.110669 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:06.110651    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwhr9g" event={"ID":"8b0ef6db-1c00-4359-bb19-6390bc20b64f","Type":"ContainerDied","Data":"1bc5f8eca348b1e060f753e65fe8f912de835485539036ce667260838fb4d3a7"}
Oct 30 18:44:06.110819 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:06.110693    4853 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bc5f8eca348b1e060f753e65fe8f912de835485539036ce667260838fb4d3a7"
Oct 30 18:44:09.461385 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:09.461318    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:10.145753 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:10.145718    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:11.499956 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.499923    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl"]
Oct 30 18:44:11.500373 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.500196    4853 cpu_manager.go:414] "containerMap: removing container" podUID="8b0ef6db-1c00-4359-bb19-6390bc20b64f" containerName="pull"
Oct 30 18:44:11.500373 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.500207    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0ef6db-1c00-4359-bb19-6390bc20b64f" containerName="pull"
Oct 30 18:44:11.500373 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.500217    4853 cpu_manager.go:414] "containerMap: removing container" podUID="8b0ef6db-1c00-4359-bb19-6390bc20b64f" containerName="extract"
Oct 30 18:44:11.500373 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.500222    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0ef6db-1c00-4359-bb19-6390bc20b64f" containerName="extract"
Oct 30 18:44:11.500373 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.500238    4853 cpu_manager.go:414] "containerMap: removing container" podUID="8b0ef6db-1c00-4359-bb19-6390bc20b64f" containerName="util"
Oct 30 18:44:11.500373 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.500243    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0ef6db-1c00-4359-bb19-6390bc20b64f" containerName="util"
Oct 30 18:44:11.500373 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.500286    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0ef6db-1c00-4359-bb19-6390bc20b64f" containerName="extract"
Oct 30 18:44:11.502307 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.502287    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl"
Oct 30 18:44:11.555100 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.555066    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl"]
Oct 30 18:44:11.601156 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.601123    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-txgg6\""
Oct 30 18:44:11.601431 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.601399    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 18:44:11.601639 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.601517    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:44:11.606741 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.606718    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f137453-1468-415b-992f-f25495fea226-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl\" (UID: \"8f137453-1468-415b-992f-f25495fea226\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl"
Oct 30 18:44:11.606834 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.606750    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq5b4\" (UniqueName: \"kubernetes.io/projected/8f137453-1468-415b-992f-f25495fea226-kube-api-access-tq5b4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl\" (UID: \"8f137453-1468-415b-992f-f25495fea226\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl"
Oct 30 18:44:11.707196 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.707169    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f137453-1468-415b-992f-f25495fea226-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl\" (UID: \"8f137453-1468-415b-992f-f25495fea226\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl"
Oct 30 18:44:11.707281 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.707202    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5b4\" (UniqueName: \"kubernetes.io/projected/8f137453-1468-415b-992f-f25495fea226-kube-api-access-tq5b4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl\" (UID: \"8f137453-1468-415b-992f-f25495fea226\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl"
Oct 30 18:44:11.707556 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.707516    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f137453-1468-415b-992f-f25495fea226-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl\" (UID: \"8f137453-1468-415b-992f-f25495fea226\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl"
Oct 30 18:44:11.765701 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.765639    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5b4\" (UniqueName: \"kubernetes.io/projected/8f137453-1468-415b-992f-f25495fea226-kube-api-access-tq5b4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl\" (UID: \"8f137453-1468-415b-992f-f25495fea226\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl"
Oct 30 18:44:11.813572 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:11.813556    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl"
Oct 30 18:44:12.235927 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:12.235895    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl"]
Oct 30 18:44:12.243761 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:44:12.243721    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f137453_1468_415b_992f_f25495fea226.slice/crio-e7a3b4fc75ed1602e34f606d6fa22f069f1650acb02888517890287859734d23 WatchSource:0}: Error finding container e7a3b4fc75ed1602e34f606d6fa22f069f1650acb02888517890287859734d23: Status 404 returned error can't find the container with id e7a3b4fc75ed1602e34f606d6fa22f069f1650acb02888517890287859734d23
Oct 30 18:44:12.958787 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:12.958754    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:13.134267 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:13.134231    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl" event={"ID":"8f137453-1468-415b-992f-f25495fea226","Type":"ContainerStarted","Data":"e7a3b4fc75ed1602e34f606d6fa22f069f1650acb02888517890287859734d23"}
Oct 30 18:44:13.268662 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:13.268590    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:15.143847 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:15.143817    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl" event={"ID":"8f137453-1468-415b-992f-f25495fea226","Type":"ContainerStarted","Data":"d37d5236269bdf9884fca74767bdfc39c8a9cc0bb55ef21b89ff42aacdf8d240"}
Oct 30 18:44:15.357656 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:15.357610    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cnwfl" podStartSLOduration=1.696833284 podStartE2EDuration="4.357595962s" totalImagesPullingTime="2.660762678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:15.311004098 +0000 UTC m=+361.481360947" watchObservedRunningTime="2026-10-30 18:44:15.357595962 +0000 UTC m=+361.527952809"
Oct 30 18:44:16.634149 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:16.634110    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:16.884035 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:16.884007    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:17.412127 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.412090    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"]
Oct 30 18:44:17.414236 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.414218    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"
Oct 30 18:44:17.432171 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.432148    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"]
Oct 30 18:44:17.472984 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.472965    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 18:44:17.473096 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.473014    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 18:44:17.473230 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.473218    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-cmmqh\""
Oct 30 18:44:17.544196 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.544170    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/672e7381-7011-482b-975f-10645e8bc897-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tdkds\" (UID: \"672e7381-7011-482b-975f-10645e8bc897\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"
Oct 30 18:44:17.544299 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.544224    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfhq\" (UniqueName: \"kubernetes.io/projected/672e7381-7011-482b-975f-10645e8bc897-kube-api-access-lmfhq\") pod \"cert-manager-webhook-5dbbbf75cc-tdkds\" (UID: \"672e7381-7011-482b-975f-10645e8bc897\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"
Oct 30 18:44:17.567047 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.567022    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:17.645176 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.645152    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfhq\" (UniqueName: \"kubernetes.io/projected/672e7381-7011-482b-975f-10645e8bc897-kube-api-access-lmfhq\") pod \"cert-manager-webhook-5dbbbf75cc-tdkds\" (UID: \"672e7381-7011-482b-975f-10645e8bc897\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"
Oct 30 18:44:17.645560 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.645194    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/672e7381-7011-482b-975f-10645e8bc897-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tdkds\" (UID: \"672e7381-7011-482b-975f-10645e8bc897\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"
Oct 30 18:44:17.702138 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.702082    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/672e7381-7011-482b-975f-10645e8bc897-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-tdkds\" (UID: \"672e7381-7011-482b-975f-10645e8bc897\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"
Oct 30 18:44:17.702224 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.702201    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfhq\" (UniqueName: \"kubernetes.io/projected/672e7381-7011-482b-975f-10645e8bc897-kube-api-access-lmfhq\") pod \"cert-manager-webhook-5dbbbf75cc-tdkds\" (UID: \"672e7381-7011-482b-975f-10645e8bc897\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"
Oct 30 18:44:17.732845 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.732822    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"
Oct 30 18:44:17.839513 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:17.839482    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:18.111104 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:18.111076    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"]
Oct 30 18:44:18.118346 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:44:18.118325    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod672e7381_7011_482b_975f_10645e8bc897.slice/crio-8617d65cca5536787f04dbd2981a4fb108281780dec062ca26ba34ec73ca2b9a WatchSource:0}: Error finding container 8617d65cca5536787f04dbd2981a4fb108281780dec062ca26ba34ec73ca2b9a: Status 404 returned error can't find the container with id 8617d65cca5536787f04dbd2981a4fb108281780dec062ca26ba34ec73ca2b9a
Oct 30 18:44:18.154978 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:18.154951    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds" event={"ID":"672e7381-7011-482b-975f-10645e8bc897","Type":"ContainerStarted","Data":"8617d65cca5536787f04dbd2981a4fb108281780dec062ca26ba34ec73ca2b9a"}
Oct 30 18:44:18.953172 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:18.953142    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:19.453476 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:19.453444    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:19.880628 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:44:19.880595    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:44:20.495027 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:20.494993    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:20.727042 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:20.727009    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:20.949485 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:20.949452    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"]
Oct 30 18:44:20.952037 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:20.952014    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:20.967435 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:20.967411    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:20.967584 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:20.967559    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjbth\" (UniqueName: \"kubernetes.io/projected/83e61f3c-869c-4954-ae2a-0d40833ad314-kube-api-access-zjbth\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:20.967700 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:20.967685    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:21.049032 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.049009    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:44:21.049208 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.049172    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:44:21.049367 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.049353    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-wdm2x\""
Oct 30 18:44:21.050377 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.050355    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"]
Oct 30 18:44:21.068591 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.068568    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:21.068713 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.068621    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:21.068713 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.068655    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjbth\" (UniqueName: \"kubernetes.io/projected/83e61f3c-869c-4954-ae2a-0d40833ad314-kube-api-access-zjbth\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:21.068980 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.068948    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:21.069076 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.068993    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:21.126134 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.126105    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjbth\" (UniqueName: \"kubernetes.io/projected/83e61f3c-869c-4954-ae2a-0d40833ad314-kube-api-access-zjbth\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:21.267341 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.267270    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:21.639459 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:21.639425    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"]
Oct 30 18:44:21.646650 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:44:21.646617    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e61f3c_869c_4954_ae2a_0d40833ad314.slice/crio-adbd452e62e3a0d8e26a39738824d834df53bac77ced181eb54a5169b0030b70 WatchSource:0}: Error finding container adbd452e62e3a0d8e26a39738824d834df53bac77ced181eb54a5169b0030b70: Status 404 returned error can't find the container with id adbd452e62e3a0d8e26a39738824d834df53bac77ced181eb54a5169b0030b70
Oct 30 18:44:22.172193 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:22.172158    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds" event={"ID":"672e7381-7011-482b-975f-10645e8bc897","Type":"ContainerStarted","Data":"a5f819228f66e91d4a3eb5adcd2b4051dfe13b52aca00be555d1e4c2129fa8ee"}
Oct 30 18:44:22.172386 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:22.172360    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"
Oct 30 18:44:22.173621 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:22.173595    4853 generic.go:356] "Generic (PLEG): container finished" podID="83e61f3c-869c-4954-ae2a-0d40833ad314" containerID="49914fb3294b5b82e163b63a91efcbc00f5fae6c1daa294e9c14026d9b786fa5" exitCode=0
Oct 30 18:44:22.173709 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:22.173678    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd" event={"ID":"83e61f3c-869c-4954-ae2a-0d40833ad314","Type":"ContainerDied","Data":"49914fb3294b5b82e163b63a91efcbc00f5fae6c1daa294e9c14026d9b786fa5"}
Oct 30 18:44:22.173759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:22.173718    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd" event={"ID":"83e61f3c-869c-4954-ae2a-0d40833ad314","Type":"ContainerStarted","Data":"adbd452e62e3a0d8e26a39738824d834df53bac77ced181eb54a5169b0030b70"}
Oct 30 18:44:22.281801 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:22.281759    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds" podStartSLOduration=1.697645948 podStartE2EDuration="5.281747495s" totalImagesPullingTime="3.584101547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:22.280828671 +0000 UTC m=+368.451185523" watchObservedRunningTime="2026-10-30 18:44:22.281747495 +0000 UTC m=+368.452104343"
Oct 30 18:44:25.924391 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:25.924344    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:26.155841 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:26.155807    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:26.190642 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:26.190561    4853 generic.go:356] "Generic (PLEG): container finished" podID="83e61f3c-869c-4954-ae2a-0d40833ad314" containerID="44bcbfc3d3eef86ee2656c5556bdcd43885cc3101d686cad4595aed4fc092357" exitCode=0
Oct 30 18:44:26.190760 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:26.190638    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd" event={"ID":"83e61f3c-869c-4954-ae2a-0d40833ad314","Type":"ContainerDied","Data":"44bcbfc3d3eef86ee2656c5556bdcd43885cc3101d686cad4595aed4fc092357"}
Oct 30 18:44:27.191000 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:27.190959    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:27.195754 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:27.195725    4853 generic.go:356] "Generic (PLEG): container finished" podID="83e61f3c-869c-4954-ae2a-0d40833ad314" containerID="7d777e3cab5ad88d1e2b955a148786871d817fc6a4cd1a7b734b99419bcbffb9" exitCode=0
Oct 30 18:44:27.195901 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:27.195769    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd" event={"ID":"83e61f3c-869c-4954-ae2a-0d40833ad314","Type":"ContainerDied","Data":"7d777e3cab5ad88d1e2b955a148786871d817fc6a4cd1a7b734b99419bcbffb9"}
Oct 30 18:44:28.180229 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.180201    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-tdkds"
Oct 30 18:44:28.329499 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.329476    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:28.421084 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.421053    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e61f3c-869c-4954-ae2a-0d40833ad314-kube-api-access-zjbth\" (UniqueName: \"kubernetes.io/projected/83e61f3c-869c-4954-ae2a-0d40833ad314-kube-api-access-zjbth\") pod \"83e61f3c-869c-4954-ae2a-0d40833ad314\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") "
Oct 30 18:44:28.421209 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.421107    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-util\" (UniqueName: \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-util\") pod \"83e61f3c-869c-4954-ae2a-0d40833ad314\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") "
Oct 30 18:44:28.421209 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.421137    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-bundle\" (UniqueName: \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-bundle\") pod \"83e61f3c-869c-4954-ae2a-0d40833ad314\" (UID: \"83e61f3c-869c-4954-ae2a-0d40833ad314\") "
Oct 30 18:44:28.421518 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.421493    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-bundle" pod "83e61f3c-869c-4954-ae2a-0d40833ad314" (UID: "83e61f3c-869c-4954-ae2a-0d40833ad314"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:28.423108 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.423079    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e61f3c-869c-4954-ae2a-0d40833ad314-kube-api-access-zjbth" pod "83e61f3c-869c-4954-ae2a-0d40833ad314" (UID: "83e61f3c-869c-4954-ae2a-0d40833ad314"). InnerVolumeSpecName "kube-api-access-zjbth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:44:28.425574 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.425553    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-util" pod "83e61f3c-869c-4954-ae2a-0d40833ad314" (UID: "83e61f3c-869c-4954-ae2a-0d40833ad314"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:44:28.476565 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.476510    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tz45x"]
Oct 30 18:44:28.476786 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.476773    4853 cpu_manager.go:414] "containerMap: removing container" podUID="83e61f3c-869c-4954-ae2a-0d40833ad314" containerName="extract"
Oct 30 18:44:28.476824 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.476786    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e61f3c-869c-4954-ae2a-0d40833ad314" containerName="extract"
Oct 30 18:44:28.476824 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.476800    4853 cpu_manager.go:414] "containerMap: removing container" podUID="83e61f3c-869c-4954-ae2a-0d40833ad314" containerName="util"
Oct 30 18:44:28.476824 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.476806    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e61f3c-869c-4954-ae2a-0d40833ad314" containerName="util"
Oct 30 18:44:28.476824 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.476823    4853 cpu_manager.go:414] "containerMap: removing container" podUID="83e61f3c-869c-4954-ae2a-0d40833ad314" containerName="pull"
Oct 30 18:44:28.476966 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.476828    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e61f3c-869c-4954-ae2a-0d40833ad314" containerName="pull"
Oct 30 18:44:28.476966 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.476867    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="83e61f3c-869c-4954-ae2a-0d40833ad314" containerName="extract"
Oct 30 18:44:28.480578 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.480563    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tz45x"
Oct 30 18:44:28.521631 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.521605    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e431167e-2c0e-4f57-87dd-0ea21d51ce98-bound-sa-token\") pod \"cert-manager-597b95cb84-tz45x\" (UID: \"e431167e-2c0e-4f57-87dd-0ea21d51ce98\") " pod="cert-manager/cert-manager-597b95cb84-tz45x"
Oct 30 18:44:28.521713 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.521666    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ls84\" (UniqueName: \"kubernetes.io/projected/e431167e-2c0e-4f57-87dd-0ea21d51ce98-kube-api-access-8ls84\") pod \"cert-manager-597b95cb84-tz45x\" (UID: \"e431167e-2c0e-4f57-87dd-0ea21d51ce98\") " pod="cert-manager/cert-manager-597b95cb84-tz45x"
Oct 30 18:44:28.521790 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.521773    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjbth\" (UniqueName: \"kubernetes.io/projected/83e61f3c-869c-4954-ae2a-0d40833ad314-kube-api-access-zjbth\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:28.521823 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.521794    4853 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-util\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:28.521823 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.521804    4853 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83e61f3c-869c-4954-ae2a-0d40833ad314-bundle\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:28.531164 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.531146    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-54vtr\""
Oct 30 18:44:28.535353 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.535333    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tz45x"]
Oct 30 18:44:28.622034 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.622012    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e431167e-2c0e-4f57-87dd-0ea21d51ce98-bound-sa-token\") pod \"cert-manager-597b95cb84-tz45x\" (UID: \"e431167e-2c0e-4f57-87dd-0ea21d51ce98\") " pod="cert-manager/cert-manager-597b95cb84-tz45x"
Oct 30 18:44:28.622117 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.622055    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ls84\" (UniqueName: \"kubernetes.io/projected/e431167e-2c0e-4f57-87dd-0ea21d51ce98-kube-api-access-8ls84\") pod \"cert-manager-597b95cb84-tz45x\" (UID: \"e431167e-2c0e-4f57-87dd-0ea21d51ce98\") " pod="cert-manager/cert-manager-597b95cb84-tz45x"
Oct 30 18:44:28.678015 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.677982    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e431167e-2c0e-4f57-87dd-0ea21d51ce98-bound-sa-token\") pod \"cert-manager-597b95cb84-tz45x\" (UID: \"e431167e-2c0e-4f57-87dd-0ea21d51ce98\") " pod="cert-manager/cert-manager-597b95cb84-tz45x"
Oct 30 18:44:28.678200 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.678182    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ls84\" (UniqueName: \"kubernetes.io/projected/e431167e-2c0e-4f57-87dd-0ea21d51ce98-kube-api-access-8ls84\") pod \"cert-manager-597b95cb84-tz45x\" (UID: \"e431167e-2c0e-4f57-87dd-0ea21d51ce98\") " pod="cert-manager/cert-manager-597b95cb84-tz45x"
Oct 30 18:44:28.791748 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:28.791701    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tz45x"
Oct 30 18:44:29.157580 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:29.157545    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tz45x"]
Oct 30 18:44:29.165220 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:44:29.165190    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode431167e_2c0e_4f57_87dd_0ea21d51ce98.slice/crio-c541f6a986e3b824a102da5553b0822ae59b28b60bd3d3f314cdd6c8f8fa4f87 WatchSource:0}: Error finding container c541f6a986e3b824a102da5553b0822ae59b28b60bd3d3f314cdd6c8f8fa4f87: Status 404 returned error can't find the container with id c541f6a986e3b824a102da5553b0822ae59b28b60bd3d3f314cdd6c8f8fa4f87
Oct 30 18:44:29.205987 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:29.205961    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd" event={"ID":"83e61f3c-869c-4954-ae2a-0d40833ad314","Type":"ContainerDied","Data":"adbd452e62e3a0d8e26a39738824d834df53bac77ced181eb54a5169b0030b70"}
Oct 30 18:44:29.206118 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:29.206006    4853 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adbd452e62e3a0d8e26a39738824d834df53bac77ced181eb54a5169b0030b70"
Oct 30 18:44:29.206118 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:29.205970    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85rsdmd"
Oct 30 18:44:29.207202 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:29.207181    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tz45x" event={"ID":"e431167e-2c0e-4f57-87dd-0ea21d51ce98","Type":"ContainerStarted","Data":"c541f6a986e3b824a102da5553b0822ae59b28b60bd3d3f314cdd6c8f8fa4f87"}
Oct 30 18:44:30.213163 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:30.213124    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tz45x" event={"ID":"e431167e-2c0e-4f57-87dd-0ea21d51ce98","Type":"ContainerStarted","Data":"b292f09a675df3faae76315be4fc638919ae67333056984c53e0b919e4d6cc57"}
Oct 30 18:44:30.323005 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:30.322946    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tz45x" podStartSLOduration=2.322931926 podStartE2EDuration="2.322931926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:30.321387228 +0000 UTC m=+376.491744076" watchObservedRunningTime="2026-10-30 18:44:30.322931926 +0000 UTC m=+376.493288774"
Oct 30 18:44:32.128235 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:32.128191    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:32.388216 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:32.388134    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:45.273044 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:45.273007    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:49.874883 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:44:49.874829    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:44:50.289370 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.289302    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"]
Oct 30 18:44:50.295434 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.295412    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.356461 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.356439    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 18:44:50.356876 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.356857    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 18:44:50.356936 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.356878    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:44:50.356936 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.356894    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:44:50.356936 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.356902    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 18:44:50.363238 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.363212    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7jf5\" (UniqueName: \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-kube-api-access-z7jf5\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.363302 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.363248    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/369aea3b-19ba-40a9-a941-884322eb7f51-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.363302 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.363276    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.363378 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.363357    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.363413 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.363378    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.363413 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.363395    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.363483 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.363411    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.401216 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.401191    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xsrfm\""
Oct 30 18:44:50.401216 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.401191    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 18:44:50.402585 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.402562    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"]
Oct 30 18:44:50.408566 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.408544    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:50.464117 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.464092    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.464201 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.464123    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.464201 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.464145    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.464201 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.464172    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.464315 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.464209    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7jf5\" (UniqueName: \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-kube-api-access-z7jf5\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.464315 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.464236    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/369aea3b-19ba-40a9-a941-884322eb7f51-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.464455 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.464434    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.464802 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.464764    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.466681 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.466655    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/369aea3b-19ba-40a9-a941-884322eb7f51-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.466782 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.466706    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.466853 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.466794    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.466914 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.466872    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.522964 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.522932    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7jf5\" (UniqueName: \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-kube-api-access-z7jf5\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.523498 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.523476    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-j4sp5\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.605676 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.605655    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:50.724963 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:50.724934    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:51.033902 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:51.033833    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"]
Oct 30 18:44:51.041184 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:44:51.041157    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369aea3b_19ba_40a9_a941_884322eb7f51.slice/crio-e6a013a287510a10c06d243122c08860e5444932cf983d28d5b56ad20afe429b WatchSource:0}: Error finding container e6a013a287510a10c06d243122c08860e5444932cf983d28d5b56ad20afe429b: Status 404 returned error can't find the container with id e6a013a287510a10c06d243122c08860e5444932cf983d28d5b56ad20afe429b
Oct 30 18:44:51.292834 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:51.292777    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5" event={"ID":"369aea3b-19ba-40a9-a941-884322eb7f51","Type":"ContainerStarted","Data":"e6a013a287510a10c06d243122c08860e5444932cf983d28d5b56ad20afe429b"}
Oct 30 18:44:51.906299 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:51.906269    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:54.261959 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:54.261918    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:54.472153 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:54.472123    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:54.695580 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:54.695519    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:55.063064 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:55.062410    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:44:55.063064 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:55.062492    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:44:55.312356 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:55.312270    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5" event={"ID":"369aea3b-19ba-40a9-a941-884322eb7f51","Type":"ContainerStarted","Data":"d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338"}
Oct 30 18:44:55.312742 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:55.312409    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:44:55.520499 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:55.520455    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5" podStartSLOduration=1.502082243 podStartE2EDuration="5.52044344s" totalImagesPullingTime="4.018361197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:55.473703562 +0000 UTC m=+401.644060411" watchObservedRunningTime="2026-10-30 18:44:55.52044344 +0000 UTC m=+401.690800287"
Oct 30 18:44:56.317861 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:56.317827    4853 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-j4sp5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:44:56.318231 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:56.317877    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5" podUID="369aea3b-19ba-40a9-a941-884322eb7f51" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:44:56.425792 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:56.425763    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:56.644205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:56.644170    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:59.317034 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:44:59.316999    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:45:00.424190 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.424154    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"]
Oct 30 18:45:00.426459 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.426441    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.478055 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.478031    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-dstgc\""
Oct 30 18:45:00.517150 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.517111    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"]
Oct 30 18:45:00.538342 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.538311    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.538449 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.538347    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.538449 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.538368    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.538585 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.538444    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.538585 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.538482    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjv6\" (UniqueName: \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-kube-api-access-9bjv6\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.538691 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.538602    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.538691 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.538652    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.538770 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.538697    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.538770 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.538738    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.576589 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.576557    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:00.639759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.639726    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.639906 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.639765    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.639906 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.639815    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.639906 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.639844    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.639906 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.639869    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.639906 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.639897    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.640123 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.639958    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjv6\" (UniqueName: \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-kube-api-access-9bjv6\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.640123 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.639996    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.640123 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.640031    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.640282 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.640189    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.640344 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.640297    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.640344 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.640323    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.640561 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.640466    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.640759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.640740    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.642199 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.642182    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.642449 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.642433    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.694822 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.694774    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjv6\" (UniqueName: \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-kube-api-access-9bjv6\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.695413 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.695391    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fdnbw\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.738396 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.738375    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:00.826295 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:00.826260    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:01.166338 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:01.166308    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"]
Oct 30 18:45:01.173419 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:45:01.173382    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9fa396_d9f2_4d45_a73d_f5c9b5a2caa9.slice/crio-0c371f82c20c2ce5ee742cdf4a9ac0f62ac802668150677faaa11b233d2749d8 WatchSource:0}: Error finding container 0c371f82c20c2ce5ee742cdf4a9ac0f62ac802668150677faaa11b233d2749d8: Status 404 returned error can't find the container with id 0c371f82c20c2ce5ee742cdf4a9ac0f62ac802668150677faaa11b233d2749d8
Oct 30 18:45:01.337357 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:01.337331    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" event={"ID":"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9","Type":"ContainerStarted","Data":"0c371f82c20c2ce5ee742cdf4a9ac0f62ac802668150677faaa11b233d2749d8"}
Oct 30 18:45:04.582625 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:04.582591    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:06.208082 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:06.208046    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:45:06.208470 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:06.208127    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:45:06.208470 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:06.208172    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:45:06.357911 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:06.357871    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" event={"ID":"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9","Type":"ContainerStarted","Data":"dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356"}
Oct 30 18:45:06.478415 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:06.478311    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" podStartSLOduration=1.446333589 podStartE2EDuration="6.478293883s" totalImagesPullingTime="5.031960294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:06.478038318 +0000 UTC m=+412.648395166" watchObservedRunningTime="2026-10-30 18:45:06.478293883 +0000 UTC m=+412.648650734"
Oct 30 18:45:06.739441 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:06.739354    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:07.744745 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:07.744704    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:08.366119 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:08.366082    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:08.367077 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:08.367053    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:45:08.789952 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:08.789878    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:09.057775 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:09.057716    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:11.013995 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.013961    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"]
Oct 30 18:45:11.017498 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.017464    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:11.113877 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.113838    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:45:11.113877 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.113836    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-2rzpv\""
Oct 30 18:45:11.113877 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.113866    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:45:11.115165 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.115145    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"]
Oct 30 18:45:11.116108 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.116087    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkmtk\" (UniqueName: \"kubernetes.io/projected/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-kube-api-access-tkmtk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:11.116187 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.116130    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:11.116228 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.116215    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:11.136131 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.136107    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"]
Oct 30 18:45:11.139200 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.139186    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:11.193480 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.193454    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-nrc6c\""
Oct 30 18:45:11.197655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.197631    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"]
Oct 30 18:45:11.216703 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.216681    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:11.216791 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.216748    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn8bk\" (UniqueName: \"kubernetes.io/projected/a480c0e4-490e-4d14-973a-95e8f0251759-kube-api-access-pn8bk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:11.216791 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.216780    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:11.216860 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.216813    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:11.216860 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.216835    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkmtk\" (UniqueName: \"kubernetes.io/projected/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-kube-api-access-tkmtk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:11.216926 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.216889    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:11.217053 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.217038    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:11.217147 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.217128    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:11.225940 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.225918    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"]
Oct 30 18:45:11.229221 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.229204    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:11.271928 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.271879    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkmtk\" (UniqueName: \"kubernetes.io/projected/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-kube-api-access-tkmtk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:11.281152 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.281132    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-4w6ct\""
Oct 30 18:45:11.286015 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.285991    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"]
Oct 30 18:45:11.304822 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.304794    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"]
Oct 30 18:45:11.307831 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.307817    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:11.317127 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.317108    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:11.317222 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.317138    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67rg5\" (UniqueName: \"kubernetes.io/projected/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-kube-api-access-67rg5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:11.317222 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.317160    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:11.317411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.317223    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:11.317411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.317261    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:11.317411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.317325    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8bk\" (UniqueName: \"kubernetes.io/projected/a480c0e4-490e-4d14-973a-95e8f0251759-kube-api-access-pn8bk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:11.317575 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.317444    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:11.317575 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.317447    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:11.334360 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.334342    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:11.359205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.359187    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-qzv5x\""
Oct 30 18:45:11.363472 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.363445    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"]
Oct 30 18:45:11.376566 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.375663    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8bk\" (UniqueName: \"kubernetes.io/projected/a480c0e4-490e-4d14-973a-95e8f0251759-kube-api-access-pn8bk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:11.417998 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.417973    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:11.418128 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.418035    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67rg5\" (UniqueName: \"kubernetes.io/projected/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-kube-api-access-67rg5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:11.418128 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.418075    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rpq\" (UniqueName: \"kubernetes.io/projected/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-kube-api-access-h4rpq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:11.418292 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.418270    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:11.418354 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.418312    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:11.418404 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.418353    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:11.418642 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.418623    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:11.418642 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.418635    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:11.454436 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.454411    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:11.476661 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.476633    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67rg5\" (UniqueName: \"kubernetes.io/projected/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-kube-api-access-67rg5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:11.519091 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.519069    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:11.519204 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.519146    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rpq\" (UniqueName: \"kubernetes.io/projected/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-kube-api-access-h4rpq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:11.519274 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.519203    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:11.519443 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.519420    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:11.519561 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.519544    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:11.543953 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.543907    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:11.574221 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.574192    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rpq\" (UniqueName: \"kubernetes.io/projected/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-kube-api-access-h4rpq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:11.622950 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.622927    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:11.706490 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.706463    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"]
Oct 30 18:45:11.712707 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:45:11.712678    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ae705b_98f0_4ec3_8fc5_6e7295c9b896.slice/crio-b2fe74282be56c2848b72e3bc97bf933ccdad66d855cd2d6717f76ae5d029332 WatchSource:0}: Error finding container b2fe74282be56c2848b72e3bc97bf933ccdad66d855cd2d6717f76ae5d029332: Status 404 returned error can't find the container with id b2fe74282be56c2848b72e3bc97bf933ccdad66d855cd2d6717f76ae5d029332
Oct 30 18:45:11.824566 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.824541    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"]
Oct 30 18:45:11.869312 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:45:11.869276    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda480c0e4_490e_4d14_973a_95e8f0251759.slice/crio-feb828a79be579cc9e74fd3d5105073a7023c1d4c1d448724a5c4ea1fd7b3052 WatchSource:0}: Error finding container feb828a79be579cc9e74fd3d5105073a7023c1d4c1d448724a5c4ea1fd7b3052: Status 404 returned error can't find the container with id feb828a79be579cc9e74fd3d5105073a7023c1d4c1d448724a5c4ea1fd7b3052
Oct 30 18:45:11.910337 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.910313    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"]
Oct 30 18:45:11.916989 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:45:11.916963    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bfbde9_d020_4663_a7cc_d5df1ed74b1e.slice/crio-3a9332606d3eeaaa754b9ce5d0a978b3b6cb8b4f1a3749dcdb59da8ca6b0217b WatchSource:0}: Error finding container 3a9332606d3eeaaa754b9ce5d0a978b3b6cb8b4f1a3749dcdb59da8ca6b0217b: Status 404 returned error can't find the container with id 3a9332606d3eeaaa754b9ce5d0a978b3b6cb8b4f1a3749dcdb59da8ca6b0217b
Oct 30 18:45:11.992880 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:11.992844    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"]
Oct 30 18:45:12.061282 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:45:12.061222    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c2b1b2_5cc7_4ec6_aaf4_783646d268c8.slice/crio-81f9e221ba6dacd0d2b81c6e6d19673eae8af005794cf0a6cc8fddf2b5ddace3 WatchSource:0}: Error finding container 81f9e221ba6dacd0d2b81c6e6d19673eae8af005794cf0a6cc8fddf2b5ddace3: Status 404 returned error can't find the container with id 81f9e221ba6dacd0d2b81c6e6d19673eae8af005794cf0a6cc8fddf2b5ddace3
Oct 30 18:45:12.385118 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.385091    4853 generic.go:356] "Generic (PLEG): container finished" podID="a480c0e4-490e-4d14-973a-95e8f0251759" containerID="b7dad36b16ad690e024efa7c92df66bd6c5537f2bab1dbf7788f145a52c4eab7" exitCode=0
Oct 30 18:45:12.385248 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.385186    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf" event={"ID":"a480c0e4-490e-4d14-973a-95e8f0251759","Type":"ContainerDied","Data":"b7dad36b16ad690e024efa7c92df66bd6c5537f2bab1dbf7788f145a52c4eab7"}
Oct 30 18:45:12.385308 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.385262    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf" event={"ID":"a480c0e4-490e-4d14-973a-95e8f0251759","Type":"ContainerStarted","Data":"feb828a79be579cc9e74fd3d5105073a7023c1d4c1d448724a5c4ea1fd7b3052"}
Oct 30 18:45:12.386722 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.386699    4853 generic.go:356] "Generic (PLEG): container finished" podID="54bfbde9-d020-4663-a7cc-d5df1ed74b1e" containerID="09ae04e6a5cd765ad44aaaa510364fae01ad82ef5d341a4d3638d03a6bd8f382" exitCode=0
Oct 30 18:45:12.386823 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.386775    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg" event={"ID":"54bfbde9-d020-4663-a7cc-d5df1ed74b1e","Type":"ContainerDied","Data":"09ae04e6a5cd765ad44aaaa510364fae01ad82ef5d341a4d3638d03a6bd8f382"}
Oct 30 18:45:12.386823 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.386818    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg" event={"ID":"54bfbde9-d020-4663-a7cc-d5df1ed74b1e","Type":"ContainerStarted","Data":"3a9332606d3eeaaa754b9ce5d0a978b3b6cb8b4f1a3749dcdb59da8ca6b0217b"}
Oct 30 18:45:12.388369 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.388343    4853 generic.go:356] "Generic (PLEG): container finished" podID="55ae705b-98f0-4ec3-8fc5-6e7295c9b896" containerID="96005e8227f0447c492d4a9eea32b18ba5228ebc8a863ccb79dff5dc2cc21567" exitCode=0
Oct 30 18:45:12.388483 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.388371    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf" event={"ID":"55ae705b-98f0-4ec3-8fc5-6e7295c9b896","Type":"ContainerDied","Data":"96005e8227f0447c492d4a9eea32b18ba5228ebc8a863ccb79dff5dc2cc21567"}
Oct 30 18:45:12.388483 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.388413    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf" event={"ID":"55ae705b-98f0-4ec3-8fc5-6e7295c9b896","Type":"ContainerStarted","Data":"b2fe74282be56c2848b72e3bc97bf933ccdad66d855cd2d6717f76ae5d029332"}
Oct 30 18:45:12.390128 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.390035    4853 generic.go:356] "Generic (PLEG): container finished" podID="15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" containerID="78d66853bcd2745d8ebb76dc69f61296a2e1843fa48339aac1b93e3685a12f6c" exitCode=0
Oct 30 18:45:12.390128 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.390077    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n" event={"ID":"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8","Type":"ContainerDied","Data":"78d66853bcd2745d8ebb76dc69f61296a2e1843fa48339aac1b93e3685a12f6c"}
Oct 30 18:45:12.390128 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:12.390115    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n" event={"ID":"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8","Type":"ContainerStarted","Data":"81f9e221ba6dacd0d2b81c6e6d19673eae8af005794cf0a6cc8fddf2b5ddace3"}
Oct 30 18:45:14.400049 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:14.400019    4853 generic.go:356] "Generic (PLEG): container finished" podID="15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" containerID="6a6e3dcf96069bee335e967522523e2c01ad65430726555bb94d3e5e77a458ad" exitCode=0
Oct 30 18:45:14.400417 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:14.400100    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n" event={"ID":"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8","Type":"ContainerDied","Data":"6a6e3dcf96069bee335e967522523e2c01ad65430726555bb94d3e5e77a458ad"}
Oct 30 18:45:14.402373 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:14.402347    4853 generic.go:356] "Generic (PLEG): container finished" podID="a480c0e4-490e-4d14-973a-95e8f0251759" containerID="9d63663e6bba3538bc3bd3e47c98df75598823832f43ba76c58da7c63ef52e33" exitCode=0
Oct 30 18:45:14.402456 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:14.402424    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf" event={"ID":"a480c0e4-490e-4d14-973a-95e8f0251759","Type":"ContainerDied","Data":"9d63663e6bba3538bc3bd3e47c98df75598823832f43ba76c58da7c63ef52e33"}
Oct 30 18:45:14.404453 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:14.404433    4853 generic.go:356] "Generic (PLEG): container finished" podID="54bfbde9-d020-4663-a7cc-d5df1ed74b1e" containerID="7f6238d7256cbfcfcd4fce3d8455219bfcdc9de89466be3fdaa37bd5d89ea12f" exitCode=0
Oct 30 18:45:14.404577 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:14.404518    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg" event={"ID":"54bfbde9-d020-4663-a7cc-d5df1ed74b1e","Type":"ContainerDied","Data":"7f6238d7256cbfcfcd4fce3d8455219bfcdc9de89466be3fdaa37bd5d89ea12f"}
Oct 30 18:45:14.406383 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:14.406363    4853 generic.go:356] "Generic (PLEG): container finished" podID="55ae705b-98f0-4ec3-8fc5-6e7295c9b896" containerID="bbde605e40d732e461738e34b39d99d2af1e84fe62d05212d4943976eed96ed9" exitCode=0
Oct 30 18:45:14.406483 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:14.406403    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf" event={"ID":"55ae705b-98f0-4ec3-8fc5-6e7295c9b896","Type":"ContainerDied","Data":"bbde605e40d732e461738e34b39d99d2af1e84fe62d05212d4943976eed96ed9"}
Oct 30 18:45:15.413281 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:15.413248    4853 generic.go:356] "Generic (PLEG): container finished" podID="15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" containerID="8ad3e0f2f854ab13e8d00dfa4c8817ac63c7fb57bd4111429ea9924516741439" exitCode=0
Oct 30 18:45:15.413659 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:15.413331    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n" event={"ID":"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8","Type":"ContainerDied","Data":"8ad3e0f2f854ab13e8d00dfa4c8817ac63c7fb57bd4111429ea9924516741439"}
Oct 30 18:45:15.415798 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:15.415778    4853 generic.go:356] "Generic (PLEG): container finished" podID="a480c0e4-490e-4d14-973a-95e8f0251759" containerID="cd7bd6659b28c37b5304060df032363984b6c302ecb2fe714f6e5cb25e9fe30d" exitCode=0
Oct 30 18:45:15.415886 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:15.415851    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf" event={"ID":"a480c0e4-490e-4d14-973a-95e8f0251759","Type":"ContainerDied","Data":"cd7bd6659b28c37b5304060df032363984b6c302ecb2fe714f6e5cb25e9fe30d"}
Oct 30 18:45:15.418262 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:15.418238    4853 generic.go:356] "Generic (PLEG): container finished" podID="54bfbde9-d020-4663-a7cc-d5df1ed74b1e" containerID="b5a9104b2f976c9082d4195d32b12439d46dd930f19a05221a721cc9a1107087" exitCode=0
Oct 30 18:45:15.418368 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:15.418315    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg" event={"ID":"54bfbde9-d020-4663-a7cc-d5df1ed74b1e","Type":"ContainerDied","Data":"b5a9104b2f976c9082d4195d32b12439d46dd930f19a05221a721cc9a1107087"}
Oct 30 18:45:15.420586 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:15.420561    4853 generic.go:356] "Generic (PLEG): container finished" podID="55ae705b-98f0-4ec3-8fc5-6e7295c9b896" containerID="e4bdccb12528f98e6367f4f4abb0378f2982ccf5b5146f60b15fb17a08fd9f05" exitCode=0
Oct 30 18:45:15.420663 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:15.420608    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf" event={"ID":"55ae705b-98f0-4ec3-8fc5-6e7295c9b896","Type":"ContainerDied","Data":"e4bdccb12528f98e6367f4f4abb0378f2982ccf5b5146f60b15fb17a08fd9f05"}
Oct 30 18:45:16.569505 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.569481    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:16.609412 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.609388    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:16.612293 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.612267    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:16.628012 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.627994    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:16.657771 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.657748    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-kube-api-access-tkmtk\" (UniqueName: \"kubernetes.io/projected/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-kube-api-access-tkmtk\") pod \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") "
Oct 30 18:45:16.657873 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.657789    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-util\" (UniqueName: \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-util\") pod \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") "
Oct 30 18:45:16.657873 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.657809    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-util\" (UniqueName: \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-util\") pod \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") "
Oct 30 18:45:16.657873 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.657830    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-kube-api-access-h4rpq\" (UniqueName: \"kubernetes.io/projected/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-kube-api-access-h4rpq\") pod \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") "
Oct 30 18:45:16.658022 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.657878    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-bundle\" (UniqueName: \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-bundle\") pod \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\" (UID: \"55ae705b-98f0-4ec3-8fc5-6e7295c9b896\") "
Oct 30 18:45:16.658022 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.657914    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-bundle\") pod \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\" (UID: \"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8\") "
Oct 30 18:45:16.659177 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.659143    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-bundle" pod "15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" (UID: "15c2b1b2-5cc7-4ec6-aaf4-783646d268c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:45:16.659583 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.659557    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-bundle" pod "55ae705b-98f0-4ec3-8fc5-6e7295c9b896" (UID: "55ae705b-98f0-4ec3-8fc5-6e7295c9b896"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:45:16.660215 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.660190    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-kube-api-access-tkmtk" pod "55ae705b-98f0-4ec3-8fc5-6e7295c9b896" (UID: "55ae705b-98f0-4ec3-8fc5-6e7295c9b896"). InnerVolumeSpecName "kube-api-access-tkmtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:16.660367 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.660346    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-kube-api-access-h4rpq" pod "15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" (UID: "15c2b1b2-5cc7-4ec6-aaf4-783646d268c8"). InnerVolumeSpecName "kube-api-access-h4rpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:16.664063 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.664039    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-util" pod "15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" (UID: "15c2b1b2-5cc7-4ec6-aaf4-783646d268c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:45:16.664116 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.664102    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-util" pod "55ae705b-98f0-4ec3-8fc5-6e7295c9b896" (UID: "55ae705b-98f0-4ec3-8fc5-6e7295c9b896"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:45:16.759084 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759030    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a480c0e4-490e-4d14-973a-95e8f0251759-kube-api-access-pn8bk\" (UniqueName: \"kubernetes.io/projected/a480c0e4-490e-4d14-973a-95e8f0251759-kube-api-access-pn8bk\") pod \"a480c0e4-490e-4d14-973a-95e8f0251759\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") "
Oct 30 18:45:16.759084 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759064    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-util\" (UniqueName: \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-util\") pod \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") "
Oct 30 18:45:16.759249 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759096    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-kube-api-access-67rg5\" (UniqueName: \"kubernetes.io/projected/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-kube-api-access-67rg5\") pod \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") "
Oct 30 18:45:16.759249 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759115    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-bundle\") pod \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\" (UID: \"54bfbde9-d020-4663-a7cc-d5df1ed74b1e\") "
Oct 30 18:45:16.759249 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759152    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-bundle\") pod \"a480c0e4-490e-4d14-973a-95e8f0251759\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") "
Oct 30 18:45:16.759249 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759190    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-util\" (UniqueName: \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-util\") pod \"a480c0e4-490e-4d14-973a-95e8f0251759\" (UID: \"a480c0e4-490e-4d14-973a-95e8f0251759\") "
Oct 30 18:45:16.759463 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759432    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkmtk\" (UniqueName: \"kubernetes.io/projected/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-kube-api-access-tkmtk\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.759463 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759457    4853 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-util\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.759589 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759472    4853 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-util\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.759589 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759485    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4rpq\" (UniqueName: \"kubernetes.io/projected/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-kube-api-access-h4rpq\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.759589 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759497    4853 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55ae705b-98f0-4ec3-8fc5-6e7295c9b896-bundle\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.759589 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759509    4853 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15c2b1b2-5cc7-4ec6-aaf4-783646d268c8-bundle\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.759947 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759688    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-bundle" pod "a480c0e4-490e-4d14-973a-95e8f0251759" (UID: "a480c0e4-490e-4d14-973a-95e8f0251759"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:45:16.759947 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.759802    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-bundle" pod "54bfbde9-d020-4663-a7cc-d5df1ed74b1e" (UID: "54bfbde9-d020-4663-a7cc-d5df1ed74b1e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:45:16.761278 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.761258    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a480c0e4-490e-4d14-973a-95e8f0251759-kube-api-access-pn8bk" pod "a480c0e4-490e-4d14-973a-95e8f0251759" (UID: "a480c0e4-490e-4d14-973a-95e8f0251759"). InnerVolumeSpecName "kube-api-access-pn8bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:16.761558 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.761541    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-kube-api-access-67rg5" pod "54bfbde9-d020-4663-a7cc-d5df1ed74b1e" (UID: "54bfbde9-d020-4663-a7cc-d5df1ed74b1e"). InnerVolumeSpecName "kube-api-access-67rg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:16.764490 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.764460    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-util" pod "54bfbde9-d020-4663-a7cc-d5df1ed74b1e" (UID: "54bfbde9-d020-4663-a7cc-d5df1ed74b1e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:45:16.764881 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.764860    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-util" pod "a480c0e4-490e-4d14-973a-95e8f0251759" (UID: "a480c0e4-490e-4d14-973a-95e8f0251759"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:45:16.861141 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.860711    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn8bk\" (UniqueName: \"kubernetes.io/projected/a480c0e4-490e-4d14-973a-95e8f0251759-kube-api-access-pn8bk\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.861141 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.860750    4853 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-util\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.861141 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.860766    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67rg5\" (UniqueName: \"kubernetes.io/projected/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-kube-api-access-67rg5\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.861141 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.860779    4853 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54bfbde9-d020-4663-a7cc-d5df1ed74b1e-bundle\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.861141 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.860797    4853 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-bundle\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:16.861141 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:16.860808    4853 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a480c0e4-490e-4d14-973a-95e8f0251759-util\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:17.432205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.432171    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n" event={"ID":"15c2b1b2-5cc7-4ec6-aaf4-783646d268c8","Type":"ContainerDied","Data":"81f9e221ba6dacd0d2b81c6e6d19673eae8af005794cf0a6cc8fddf2b5ddace3"}
Oct 30 18:45:17.432390 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.432223    4853 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81f9e221ba6dacd0d2b81c6e6d19673eae8af005794cf0a6cc8fddf2b5ddace3"
Oct 30 18:45:17.432390 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.432290    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zg5n"
Oct 30 18:45:17.434711 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.434671    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf" event={"ID":"a480c0e4-490e-4d14-973a-95e8f0251759","Type":"ContainerDied","Data":"feb828a79be579cc9e74fd3d5105073a7023c1d4c1d448724a5c4ea1fd7b3052"}
Oct 30 18:45:17.434867 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.434724    4853 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feb828a79be579cc9e74fd3d5105073a7023c1d4c1d448724a5c4ea1fd7b3052"
Oct 30 18:45:17.434867 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.434691    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c669rqf"
Oct 30 18:45:17.436993 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.436963    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg" event={"ID":"54bfbde9-d020-4663-a7cc-d5df1ed74b1e","Type":"ContainerDied","Data":"3a9332606d3eeaaa754b9ce5d0a978b3b6cb8b4f1a3749dcdb59da8ca6b0217b"}
Oct 30 18:45:17.437153 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.437018    4853 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a9332606d3eeaaa754b9ce5d0a978b3b6cb8b4f1a3749dcdb59da8ca6b0217b"
Oct 30 18:45:17.437153 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.436974    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdxxpg"
Oct 30 18:45:17.439087 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.439058    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf" event={"ID":"55ae705b-98f0-4ec3-8fc5-6e7295c9b896","Type":"ContainerDied","Data":"b2fe74282be56c2848b72e3bc97bf933ccdad66d855cd2d6717f76ae5d029332"}
Oct 30 18:45:17.439226 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.439099    4853 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2fe74282be56c2848b72e3bc97bf933ccdad66d855cd2d6717f76ae5d029332"
Oct 30 18:45:17.439226 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:17.439110    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an79sf"
Oct 30 18:45:19.873366 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:45:19.873328    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:45:21.495203 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:21.495168    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:22.313276 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:22.313239    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:23.102236 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:23.102188    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:24.332044 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:24.331994    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:26.202846 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.202807    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"]
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203078    4853 cpu_manager.go:414] "containerMap: removing container" podUID="a480c0e4-490e-4d14-973a-95e8f0251759" containerName="util"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203089    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a480c0e4-490e-4d14-973a-95e8f0251759" containerName="util"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203109    4853 cpu_manager.go:414] "containerMap: removing container" podUID="54bfbde9-d020-4663-a7cc-d5df1ed74b1e" containerName="extract"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203114    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bfbde9-d020-4663-a7cc-d5df1ed74b1e" containerName="extract"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203121    4853 cpu_manager.go:414] "containerMap: removing container" podUID="55ae705b-98f0-4ec3-8fc5-6e7295c9b896" containerName="extract"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203126    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ae705b-98f0-4ec3-8fc5-6e7295c9b896" containerName="extract"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203135    4853 cpu_manager.go:414] "containerMap: removing container" podUID="15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" containerName="pull"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203140    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" containerName="pull"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203150    4853 cpu_manager.go:414] "containerMap: removing container" podUID="a480c0e4-490e-4d14-973a-95e8f0251759" containerName="extract"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203154    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a480c0e4-490e-4d14-973a-95e8f0251759" containerName="extract"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203162    4853 cpu_manager.go:414] "containerMap: removing container" podUID="55ae705b-98f0-4ec3-8fc5-6e7295c9b896" containerName="util"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203166    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ae705b-98f0-4ec3-8fc5-6e7295c9b896" containerName="util"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203172    4853 cpu_manager.go:414] "containerMap: removing container" podUID="15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" containerName="util"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203176    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" containerName="util"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203183    4853 cpu_manager.go:414] "containerMap: removing container" podUID="54bfbde9-d020-4663-a7cc-d5df1ed74b1e" containerName="pull"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203187    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bfbde9-d020-4663-a7cc-d5df1ed74b1e" containerName="pull"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203193    4853 cpu_manager.go:414] "containerMap: removing container" podUID="15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" containerName="extract"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203197    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" containerName="extract"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203205    4853 cpu_manager.go:414] "containerMap: removing container" podUID="54bfbde9-d020-4663-a7cc-d5df1ed74b1e" containerName="util"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203210    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bfbde9-d020-4663-a7cc-d5df1ed74b1e" containerName="util"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203216    4853 cpu_manager.go:414] "containerMap: removing container" podUID="a480c0e4-490e-4d14-973a-95e8f0251759" containerName="pull"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203220    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a480c0e4-490e-4d14-973a-95e8f0251759" containerName="pull"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203226    4853 cpu_manager.go:414] "containerMap: removing container" podUID="55ae705b-98f0-4ec3-8fc5-6e7295c9b896" containerName="pull"
Oct 30 18:45:26.203257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203230    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ae705b-98f0-4ec3-8fc5-6e7295c9b896" containerName="pull"
Oct 30 18:45:26.204013 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203274    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bfbde9-d020-4663-a7cc-d5df1ed74b1e" containerName="extract"
Oct 30 18:45:26.204013 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203281    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ae705b-98f0-4ec3-8fc5-6e7295c9b896" containerName="extract"
Oct 30 18:45:26.204013 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203288    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="a480c0e4-490e-4d14-973a-95e8f0251759" containerName="extract"
Oct 30 18:45:26.204013 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.203294    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c2b1b2-5cc7-4ec6-aaf4-783646d268c8" containerName="extract"
Oct 30 18:45:26.206330 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.206309    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:26.246287 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.246254    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"]
Oct 30 18:45:26.291328 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.291303    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 18:45:26.291440 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.291354    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 18:45:26.291440 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.291396    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-72xqt\""
Oct 30 18:45:26.328318 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.328300    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhm6q\" (UniqueName: \"kubernetes.io/projected/cd78d441-3f8a-4b36-a98c-1083914762bf-kube-api-access-xhm6q\") pod \"kuadrant-operator-controller-manager-58559b5d46-bmrjw\" (UID: \"cd78d441-3f8a-4b36-a98c-1083914762bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:26.328430 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.328329    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd78d441-3f8a-4b36-a98c-1083914762bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58559b5d46-bmrjw\" (UID: \"cd78d441-3f8a-4b36-a98c-1083914762bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:26.429580 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.429554    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm6q\" (UniqueName: \"kubernetes.io/projected/cd78d441-3f8a-4b36-a98c-1083914762bf-kube-api-access-xhm6q\") pod \"kuadrant-operator-controller-manager-58559b5d46-bmrjw\" (UID: \"cd78d441-3f8a-4b36-a98c-1083914762bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:26.429690 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.429586    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd78d441-3f8a-4b36-a98c-1083914762bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58559b5d46-bmrjw\" (UID: \"cd78d441-3f8a-4b36-a98c-1083914762bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:26.429933 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.429918    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd78d441-3f8a-4b36-a98c-1083914762bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58559b5d46-bmrjw\" (UID: \"cd78d441-3f8a-4b36-a98c-1083914762bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:26.488852 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.488790    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhm6q\" (UniqueName: \"kubernetes.io/projected/cd78d441-3f8a-4b36-a98c-1083914762bf-kube-api-access-xhm6q\") pod \"kuadrant-operator-controller-manager-58559b5d46-bmrjw\" (UID: \"cd78d441-3f8a-4b36-a98c-1083914762bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:26.489735 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.489711    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:26.519646 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.519624    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:26.909039 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:26.909006    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:27.007408 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:27.007383    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"]
Oct 30 18:45:27.064625 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:45:27.064600    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd78d441_3f8a_4b36_a98c_1083914762bf.slice/crio-fcc6ff8e6a41128cdc0051072bdb8b16b835470c5f93c909273882d745bbc7d4 WatchSource:0}: Error finding container fcc6ff8e6a41128cdc0051072bdb8b16b835470c5f93c909273882d745bbc7d4: Status 404 returned error can't find the container with id fcc6ff8e6a41128cdc0051072bdb8b16b835470c5f93c909273882d745bbc7d4
Oct 30 18:45:27.486079 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:27.486050    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw" event={"ID":"cd78d441-3f8a-4b36-a98c-1083914762bf","Type":"ContainerStarted","Data":"fcc6ff8e6a41128cdc0051072bdb8b16b835470c5f93c909273882d745bbc7d4"}
Oct 30 18:45:27.551491 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:27.551464    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:29.498686 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:29.498646    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:29.739839 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:29.739808    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:30.757393 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:30.757358    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:30.967757 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:30.967715    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:31.153490 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:31.153457    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp"]
Oct 30 18:45:31.156940 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:31.156909    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp"
Oct 30 18:45:31.254790 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:31.254764    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jwjq9\""
Oct 30 18:45:31.256250 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:31.256226    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp"]
Oct 30 18:45:31.267166 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:31.267138    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5zb\" (UniqueName: \"kubernetes.io/projected/4b0542b1-baff-4f8d-803a-a7760bc028cd-kube-api-access-6z5zb\") pod \"limitador-operator-controller-manager-69b7bb8c5c-5b5dp\" (UID: \"4b0542b1-baff-4f8d-803a-a7760bc028cd\") " pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp"
Oct 30 18:45:31.368139 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:31.368110    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5zb\" (UniqueName: \"kubernetes.io/projected/4b0542b1-baff-4f8d-803a-a7760bc028cd-kube-api-access-6z5zb\") pod \"limitador-operator-controller-manager-69b7bb8c5c-5b5dp\" (UID: \"4b0542b1-baff-4f8d-803a-a7760bc028cd\") " pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp"
Oct 30 18:45:31.429549 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:31.429466    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5zb\" (UniqueName: \"kubernetes.io/projected/4b0542b1-baff-4f8d-803a-a7760bc028cd-kube-api-access-6z5zb\") pod \"limitador-operator-controller-manager-69b7bb8c5c-5b5dp\" (UID: \"4b0542b1-baff-4f8d-803a-a7760bc028cd\") " pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp"
Oct 30 18:45:31.471241 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:31.471216    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp"
Oct 30 18:45:31.758086 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:31.758059    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:31.877116 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:31.877090    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp"]
Oct 30 18:45:31.898888 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:45:31.898863    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0542b1_baff_4f8d_803a_a7760bc028cd.slice/crio-8e5bde6d0625699e1b663c89d72517bed943eade308a3d0f8ca0ed4dcdfb294e WatchSource:0}: Error finding container 8e5bde6d0625699e1b663c89d72517bed943eade308a3d0f8ca0ed4dcdfb294e: Status 404 returned error can't find the container with id 8e5bde6d0625699e1b663c89d72517bed943eade308a3d0f8ca0ed4dcdfb294e
Oct 30 18:45:32.512763 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:32.512730    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw" event={"ID":"cd78d441-3f8a-4b36-a98c-1083914762bf","Type":"ContainerStarted","Data":"4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb"}
Oct 30 18:45:32.513046 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:32.512795    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:32.513973 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:32.513950    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp" event={"ID":"4b0542b1-baff-4f8d-803a-a7760bc028cd","Type":"ContainerStarted","Data":"8e5bde6d0625699e1b663c89d72517bed943eade308a3d0f8ca0ed4dcdfb294e"}
Oct 30 18:45:32.727863 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:32.727814    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw" podStartSLOduration=2.055618199 podStartE2EDuration="6.727800224s" totalImagesPullingTime="4.672182025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:32.726291233 +0000 UTC m=+438.896648082" watchObservedRunningTime="2026-10-30 18:45:32.727800224 +0000 UTC m=+438.898157127"
Oct 30 18:45:32.796472 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:32.796404    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:35.526936 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:35.526904    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp" event={"ID":"4b0542b1-baff-4f8d-803a-a7760bc028cd","Type":"ContainerStarted","Data":"93b53b5391280972ceeffc4f9e967a904bf39e7cbdac2477863206363e706160"}
Oct 30 18:45:35.527398 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:35.527032    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp"
Oct 30 18:45:35.639453 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:35.639403    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp" podStartSLOduration=1.847519255 podStartE2EDuration="4.639389688s" totalImagesPullingTime="2.791870433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:35.63772701 +0000 UTC m=+441.808083860" watchObservedRunningTime="2026-10-30 18:45:35.639389688 +0000 UTC m=+441.809746536"
Oct 30 18:45:36.893084 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:36.893039    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:37.099444 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:37.099405    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:37.311492 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:37.311415    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:37.521887 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:37.521860    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:37.798981 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:37.798952    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:43.521192 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:43.521157    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:44.164199 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:44.164171    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"]
Oct 30 18:45:44.171503 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:44.171482    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"
Oct 30 18:45:44.237654 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:44.237630    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"]
Oct 30 18:45:44.270254 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:44.270227    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae793fdf-7781-4e49-ad45-5bec61829d41-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-658c459c99-dk64c\" (UID: \"ae793fdf-7781-4e49-ad45-5bec61829d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"
Oct 30 18:45:44.270353 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:44.270266    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbp57\" (UniqueName: \"kubernetes.io/projected/ae793fdf-7781-4e49-ad45-5bec61829d41-kube-api-access-pbp57\") pod \"kuadrant-operator-controller-manager-658c459c99-dk64c\" (UID: \"ae793fdf-7781-4e49-ad45-5bec61829d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"
Oct 30 18:45:44.371215 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:44.371188    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae793fdf-7781-4e49-ad45-5bec61829d41-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-658c459c99-dk64c\" (UID: \"ae793fdf-7781-4e49-ad45-5bec61829d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"
Oct 30 18:45:44.371340 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:44.371232    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbp57\" (UniqueName: \"kubernetes.io/projected/ae793fdf-7781-4e49-ad45-5bec61829d41-kube-api-access-pbp57\") pod \"kuadrant-operator-controller-manager-658c459c99-dk64c\" (UID: \"ae793fdf-7781-4e49-ad45-5bec61829d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"
Oct 30 18:45:44.371594 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:44.371575    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae793fdf-7781-4e49-ad45-5bec61829d41-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-658c459c99-dk64c\" (UID: \"ae793fdf-7781-4e49-ad45-5bec61829d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"
Oct 30 18:45:44.431977 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:44.431921    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbp57\" (UniqueName: \"kubernetes.io/projected/ae793fdf-7781-4e49-ad45-5bec61829d41-kube-api-access-pbp57\") pod \"kuadrant-operator-controller-manager-658c459c99-dk64c\" (UID: \"ae793fdf-7781-4e49-ad45-5bec61829d41\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"
Oct 30 18:45:44.484590 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:44.484563    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"
Oct 30 18:45:44.910733 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:44.910707    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"]
Oct 30 18:45:44.918737 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:45:44.918708    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae793fdf_7781_4e49_ad45_5bec61829d41.slice/crio-e41a23e04ba651dd477cd02e6d041202e54a436b05ee4b33e1eacf0167f2610f WatchSource:0}: Error finding container e41a23e04ba651dd477cd02e6d041202e54a436b05ee4b33e1eacf0167f2610f: Status 404 returned error can't find the container with id e41a23e04ba651dd477cd02e6d041202e54a436b05ee4b33e1eacf0167f2610f
Oct 30 18:45:45.577709 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:45.577673    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c" event={"ID":"ae793fdf-7781-4e49-ad45-5bec61829d41","Type":"ContainerStarted","Data":"141730b07131389d790890ca1a64369788a87d004c586f09332515126f4acaac"}
Oct 30 18:45:45.577912 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:45.577729    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c" event={"ID":"ae793fdf-7781-4e49-ad45-5bec61829d41","Type":"ContainerStarted","Data":"e41a23e04ba651dd477cd02e6d041202e54a436b05ee4b33e1eacf0167f2610f"}
Oct 30 18:45:45.577912 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:45.577790    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"
Oct 30 18:45:45.723411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:45.723380    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:45.731870 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:45.731829    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c" podStartSLOduration=1.731816217 podStartE2EDuration="1.731816217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:45.730373863 +0000 UTC m=+451.900730712" watchObservedRunningTime="2026-10-30 18:45:45.731816217 +0000 UTC m=+451.902173065"
Oct 30 18:45:45.960364 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:45.960295    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:46.539113 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:46.539084    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-5b5dp"
Oct 30 18:45:48.710223 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:48.710184    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:48.921303 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:48.921261    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:49.873928 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:45:49.873892    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:45:50.892462 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:50.892417    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:51.207827 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:51.207752    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:51.532842 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:51.532762    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:56.584960 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:56.584920    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-dk64c"
Oct 30 18:45:56.793250 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:56.793224    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"]
Oct 30 18:45:56.793641 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:56.793618    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw" podUID="cd78d441-3f8a-4b36-a98c-1083914762bf" containerName="manager" containerID="cri-o://4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb" gracePeriod=10
Oct 30 18:45:57.030970 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.030948    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:57.167451 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.167383    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd78d441-3f8a-4b36-a98c-1083914762bf-kube-api-access-xhm6q\" (UniqueName: \"kubernetes.io/projected/cd78d441-3f8a-4b36-a98c-1083914762bf-kube-api-access-xhm6q\") pod \"cd78d441-3f8a-4b36-a98c-1083914762bf\" (UID: \"cd78d441-3f8a-4b36-a98c-1083914762bf\") "
Oct 30 18:45:57.167611 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.167469    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd78d441-3f8a-4b36-a98c-1083914762bf-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd78d441-3f8a-4b36-a98c-1083914762bf-extensions-socket-volume\") pod \"cd78d441-3f8a-4b36-a98c-1083914762bf\" (UID: \"cd78d441-3f8a-4b36-a98c-1083914762bf\") "
Oct 30 18:45:57.167935 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.167911    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd78d441-3f8a-4b36-a98c-1083914762bf-extensions-socket-volume" pod "cd78d441-3f8a-4b36-a98c-1083914762bf" (UID: "cd78d441-3f8a-4b36-a98c-1083914762bf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:45:57.169684 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.169662    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd78d441-3f8a-4b36-a98c-1083914762bf-kube-api-access-xhm6q" pod "cd78d441-3f8a-4b36-a98c-1083914762bf" (UID: "cd78d441-3f8a-4b36-a98c-1083914762bf"). InnerVolumeSpecName "kube-api-access-xhm6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:57.268160 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.268136    4853 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd78d441-3f8a-4b36-a98c-1083914762bf-extensions-socket-volume\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:57.268160 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.268156    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhm6q\" (UniqueName: \"kubernetes.io/projected/cd78d441-3f8a-4b36-a98c-1083914762bf-kube-api-access-xhm6q\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:57.637958 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.637926    4853 generic.go:356] "Generic (PLEG): container finished" podID="cd78d441-3f8a-4b36-a98c-1083914762bf" containerID="4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb" exitCode=0
Oct 30 18:45:57.638294 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.637983    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"
Oct 30 18:45:57.638294 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.638011    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw" event={"ID":"cd78d441-3f8a-4b36-a98c-1083914762bf","Type":"ContainerDied","Data":"4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb"}
Oct 30 18:45:57.638294 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.638089    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw" event={"ID":"cd78d441-3f8a-4b36-a98c-1083914762bf","Type":"ContainerDied","Data":"fcc6ff8e6a41128cdc0051072bdb8b16b835470c5f93c909273882d745bbc7d4"}
Oct 30 18:45:57.638294 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.638113    4853 scope.go:122] "RemoveContainer" containerID="4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb"
Oct 30 18:45:57.646705 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.646688    4853 scope.go:122] "RemoveContainer" containerID="4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb"
Oct 30 18:45:57.646980 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:45:57.646961    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb\": container with ID starting with 4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb not found: ID does not exist" containerID="4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb"
Oct 30 18:45:57.647035 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.646996    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb"} err="failed to get container status \"4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb\": rpc error: code = NotFound desc = could not find container \"4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb\": container with ID starting with 4df1cf8e5ad13fdc24c80666c22857cfd8d7c97db4c280ea37e83f80843e30eb not found: ID does not exist"
Oct 30 18:45:57.797735 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.797712    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"]
Oct 30 18:45:57.844422 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.844396    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58559b5d46-bmrjw"]
Oct 30 18:45:57.938993 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:57.938935    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd78d441-3f8a-4b36-a98c-1083914762bf" path="/var/lib/kubelet/pods/cd78d441-3f8a-4b36-a98c-1083914762bf/volumes"
Oct 30 18:45:58.113738 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:58.113707    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:58.354114 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:58.354066    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:58.571482 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:58.571455    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:58.772687 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:45:58.772597    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:07.544121 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:07.544057    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:07.783382 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:07.783344    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:10.524563 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:10.524476    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:10.725974 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:10.725939    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:11.628065 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:11.628030    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:11.828908 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:11.828877    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:13.035819 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:13.035784    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:13.283607 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:13.283573    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:19.874872 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:46:19.874833    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:46:34.877053 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:34.877018    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:35.126968 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:35.126937    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:35.342112 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:35.342083    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:36.312118 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:36.312078    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:36.520764 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:36.520729    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:38.861074 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:38.861043    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cn92d"]
Oct 30 18:46:38.861574 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:38.861328    4853 cpu_manager.go:414] "containerMap: removing container" podUID="cd78d441-3f8a-4b36-a98c-1083914762bf" containerName="manager"
Oct 30 18:46:38.861574 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:38.861339    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd78d441-3f8a-4b36-a98c-1083914762bf" containerName="manager"
Oct 30 18:46:38.861574 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:38.861410    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd78d441-3f8a-4b36-a98c-1083914762bf" containerName="manager"
Oct 30 18:46:38.863322 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:38.863306    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cn92d"
Oct 30 18:46:38.917034 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:38.917010    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-69q5d\""
Oct 30 18:46:38.921487 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:38.921461    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cn92d"]
Oct 30 18:46:38.932457 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:38.932438    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drl4k\" (UniqueName: \"kubernetes.io/projected/9e1b2f45-5f7a-40cd-9628-c441c01d6b52-kube-api-access-drl4k\") pod \"authorino-56c6c9ddd5-cn92d\" (UID: \"9e1b2f45-5f7a-40cd-9628-c441c01d6b52\") " pod="kuadrant-system/authorino-56c6c9ddd5-cn92d"
Oct 30 18:46:39.032931 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:39.032910    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drl4k\" (UniqueName: \"kubernetes.io/projected/9e1b2f45-5f7a-40cd-9628-c441c01d6b52-kube-api-access-drl4k\") pod \"authorino-56c6c9ddd5-cn92d\" (UID: \"9e1b2f45-5f7a-40cd-9628-c441c01d6b52\") " pod="kuadrant-system/authorino-56c6c9ddd5-cn92d"
Oct 30 18:46:39.092333 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:39.092312    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drl4k\" (UniqueName: \"kubernetes.io/projected/9e1b2f45-5f7a-40cd-9628-c441c01d6b52-kube-api-access-drl4k\") pod \"authorino-56c6c9ddd5-cn92d\" (UID: \"9e1b2f45-5f7a-40cd-9628-c441c01d6b52\") " pod="kuadrant-system/authorino-56c6c9ddd5-cn92d"
Oct 30 18:46:39.174020 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:39.173975    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cn92d"
Oct 30 18:46:39.537690 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:39.537615    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cn92d"]
Oct 30 18:46:39.544937 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:46:39.544901    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e1b2f45_5f7a_40cd_9628_c441c01d6b52.slice/crio-5bca8a6eca7439380b7a473c427cb8559a79dcaa5089eaa59c74afb0da5ad252 WatchSource:0}: Error finding container 5bca8a6eca7439380b7a473c427cb8559a79dcaa5089eaa59c74afb0da5ad252: Status 404 returned error can't find the container with id 5bca8a6eca7439380b7a473c427cb8559a79dcaa5089eaa59c74afb0da5ad252
Oct 30 18:46:39.831737 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:39.831691    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cn92d" event={"ID":"9e1b2f45-5f7a-40cd-9628-c441c01d6b52","Type":"ContainerStarted","Data":"5bca8a6eca7439380b7a473c427cb8559a79dcaa5089eaa59c74afb0da5ad252"}
Oct 30 18:46:44.857488 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:44.857454    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cn92d" event={"ID":"9e1b2f45-5f7a-40cd-9628-c441c01d6b52","Type":"ContainerStarted","Data":"b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14"}
Oct 30 18:46:44.965497 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:44.965452    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-cn92d" podStartSLOduration=2.3275494070000002 podStartE2EDuration="6.965439187s" totalImagesPullingTime="4.63788978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:44.964400476 +0000 UTC m=+511.134757323" watchObservedRunningTime="2026-10-30 18:46:44.965439187 +0000 UTC m=+511.135796036"
Oct 30 18:46:49.874613 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:46:49.874574    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:46:51.272441 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:51.272397    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:51.485284 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:46:51.485254    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:01.495252 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:01.495213    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:01.789850 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:01.789775    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:09.439643 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.439576    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cn92d"]
Oct 30 18:47:09.440043 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.439794    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-cn92d" podUID="9e1b2f45-5f7a-40cd-9628-c441c01d6b52" containerName="authorino" containerID="cri-o://b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14" gracePeriod=30
Oct 30 18:47:09.677428 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.677403    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cn92d"
Oct 30 18:47:09.744640 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.744588    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1b2f45-5f7a-40cd-9628-c441c01d6b52-kube-api-access-drl4k\" (UniqueName: \"kubernetes.io/projected/9e1b2f45-5f7a-40cd-9628-c441c01d6b52-kube-api-access-drl4k\") pod \"9e1b2f45-5f7a-40cd-9628-c441c01d6b52\" (UID: \"9e1b2f45-5f7a-40cd-9628-c441c01d6b52\") "
Oct 30 18:47:09.746618 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.746588    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1b2f45-5f7a-40cd-9628-c441c01d6b52-kube-api-access-drl4k" pod "9e1b2f45-5f7a-40cd-9628-c441c01d6b52" (UID: "9e1b2f45-5f7a-40cd-9628-c441c01d6b52"). InnerVolumeSpecName "kube-api-access-drl4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:09.845663 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.845638    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drl4k\" (UniqueName: \"kubernetes.io/projected/9e1b2f45-5f7a-40cd-9628-c441c01d6b52-kube-api-access-drl4k\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:09.975554 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.975513    4853 generic.go:356] "Generic (PLEG): container finished" podID="9e1b2f45-5f7a-40cd-9628-c441c01d6b52" containerID="b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14" exitCode=0
Oct 30 18:47:09.975715 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.975571    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cn92d"
Oct 30 18:47:09.975715 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.975592    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cn92d" event={"ID":"9e1b2f45-5f7a-40cd-9628-c441c01d6b52","Type":"ContainerDied","Data":"b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14"}
Oct 30 18:47:09.975715 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.975634    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cn92d" event={"ID":"9e1b2f45-5f7a-40cd-9628-c441c01d6b52","Type":"ContainerDied","Data":"5bca8a6eca7439380b7a473c427cb8559a79dcaa5089eaa59c74afb0da5ad252"}
Oct 30 18:47:09.975715 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.975651    4853 scope.go:122] "RemoveContainer" containerID="b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14"
Oct 30 18:47:09.983964 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.983949    4853 scope.go:122] "RemoveContainer" containerID="b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14"
Oct 30 18:47:09.984241 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:47:09.984221    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14\": container with ID starting with b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14 not found: ID does not exist" containerID="b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14"
Oct 30 18:47:09.984321 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:09.984251    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14"} err="failed to get container status \"b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14\": rpc error: code = NotFound desc = could not find container \"b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14\": container with ID starting with b2491aca7bb82168e5871dbaf69723e7fc1138f70afbb84f9b4f52062727ef14 not found: ID does not exist"
Oct 30 18:47:10.131450 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:10.131422    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cn92d"]
Oct 30 18:47:10.134442 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:10.134422    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cn92d"]
Oct 30 18:47:11.785425 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:11.785391    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:11.939338 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:11.939298    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1b2f45-5f7a-40cd-9628-c441c01d6b52" path="/var/lib/kubelet/pods/9e1b2f45-5f7a-40cd-9628-c441c01d6b52/volumes"
Oct 30 18:47:12.009115 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:12.009090    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:19.874733 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:47:19.874690    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:47:28.516848 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.516806    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:28.607202 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.607165    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"]
Oct 30 18:47:28.607476 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.607462    4853 cpu_manager.go:414] "containerMap: removing container" podUID="9e1b2f45-5f7a-40cd-9628-c441c01d6b52" containerName="authorino"
Oct 30 18:47:28.607521 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.607476    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1b2f45-5f7a-40cd-9628-c441c01d6b52" containerName="authorino"
Oct 30 18:47:28.607595 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.607562    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1b2f45-5f7a-40cd-9628-c441c01d6b52" containerName="authorino"
Oct 30 18:47:28.610548 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.610511    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.670613 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.670582    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgr7q\" (UniqueName: \"kubernetes.io/projected/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-kube-api-access-kgr7q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.670613 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.670614    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.670768 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.670635    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.670768 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.670699    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.670768 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.670758    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.670861 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.670786    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.670861 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.670818    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.700520 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.700491    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"]
Oct 30 18:47:28.771872 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.771814    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.771872 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.771842    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.771872 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.771862    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.772089 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.771887    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.772089 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.771909    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgr7q\" (UniqueName: \"kubernetes.io/projected/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-kube-api-access-kgr7q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.772089 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.771928    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.772089 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.771950    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.772963 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.772927    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.774475 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.774449    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.774590 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.774501    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.774652 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.774634    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.774689 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.774656    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.827337 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.827315    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgr7q\" (UniqueName: \"kubernetes.io/projected/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-kube-api-access-kgr7q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.827478 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.827460    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbc10ed-835f-4b67-a55a-4bae7bf054ab-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kpb9g\" (UID: \"7cbc10ed-835f-4b67-a55a-4bae7bf054ab\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:28.923134 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:28.923110    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:29.345745 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:29.345708    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"]
Oct 30 18:47:29.353051 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:47:29.353020    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbc10ed_835f_4b67_a55a_4bae7bf054ab.slice/crio-ae3bd4a1683374c1ff92ecc42ebcf0cdd29cd4c5f4b42913fd95bda2c65e641c WatchSource:0}: Error finding container ae3bd4a1683374c1ff92ecc42ebcf0cdd29cd4c5f4b42913fd95bda2c65e641c: Status 404 returned error can't find the container with id ae3bd4a1683374c1ff92ecc42ebcf0cdd29cd4c5f4b42913fd95bda2c65e641c
Oct 30 18:47:29.358072 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:29.358031    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:29.358184 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:29.358111    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:47:30.034209 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:30.034171    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:30.065028 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:30.064991    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g" event={"ID":"7cbc10ed-835f-4b67-a55a-4bae7bf054ab","Type":"ContainerStarted","Data":"75c0bb8c07e5ad812f60db439e399a4a036daa01bbf00bcf029413314dd522f3"}
Oct 30 18:47:30.065183 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:30.065054    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g" event={"ID":"7cbc10ed-835f-4b67-a55a-4bae7bf054ab","Type":"ContainerStarted","Data":"ae3bd4a1683374c1ff92ecc42ebcf0cdd29cd4c5f4b42913fd95bda2c65e641c"}
Oct 30 18:47:30.065264 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:30.065183    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:30.066720 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:30.066695    4853 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kpb9g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:47:30.066825 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:30.066749    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g" podUID="7cbc10ed-835f-4b67-a55a-4bae7bf054ab" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:47:30.260206 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:30.260163    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g" podStartSLOduration=2.260152477 podStartE2EDuration="2.260152477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:30.253010572 +0000 UTC m=+556.423367421" watchObservedRunningTime="2026-10-30 18:47:30.260152477 +0000 UTC m=+556.430509324"
Oct 30 18:47:31.069601 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.069572    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kpb9g"
Oct 30 18:47:31.283021 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.282983    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"]
Oct 30 18:47:31.283206 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.283165    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5" podUID="369aea3b-19ba-40a9-a941-884322eb7f51" containerName="discovery" containerID="cri-o://d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338" gracePeriod=30
Oct 30 18:47:31.526768 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.526740    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:47:31.592129 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.592104    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-ca-configmap\") pod \"369aea3b-19ba-40a9-a941-884322eb7f51\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") "
Oct 30 18:47:31.592270 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.592145    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-istio-token\" (UniqueName: \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-istio-token\") pod \"369aea3b-19ba-40a9-a941-884322eb7f51\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") "
Oct 30 18:47:31.592270 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.592180    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-kubeconfig\") pod \"369aea3b-19ba-40a9-a941-884322eb7f51\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") "
Oct 30 18:47:31.592270 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.592219    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369aea3b-19ba-40a9-a941-884322eb7f51-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/369aea3b-19ba-40a9-a941-884322eb7f51-local-certs\") pod \"369aea3b-19ba-40a9-a941-884322eb7f51\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") "
Oct 30 18:47:31.592270 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.592244    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-kube-api-access-z7jf5\" (UniqueName: \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-kube-api-access-z7jf5\") pod \"369aea3b-19ba-40a9-a941-884322eb7f51\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") "
Oct 30 18:47:31.592481 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.592270    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-dns-cert\") pod \"369aea3b-19ba-40a9-a941-884322eb7f51\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") "
Oct 30 18:47:31.592481 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.592317    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-cacerts\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-cacerts\") pod \"369aea3b-19ba-40a9-a941-884322eb7f51\" (UID: \"369aea3b-19ba-40a9-a941-884322eb7f51\") "
Oct 30 18:47:31.592616 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.592567    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-ca-configmap" pod "369aea3b-19ba-40a9-a941-884322eb7f51" (UID: "369aea3b-19ba-40a9-a941-884322eb7f51"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:31.595206 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.595178    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-istio-token" pod "369aea3b-19ba-40a9-a941-884322eb7f51" (UID: "369aea3b-19ba-40a9-a941-884322eb7f51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:31.595206 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.595178    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369aea3b-19ba-40a9-a941-884322eb7f51-local-certs" pod "369aea3b-19ba-40a9-a941-884322eb7f51" (UID: "369aea3b-19ba-40a9-a941-884322eb7f51"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:47:31.595408 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.595386    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-kubeconfig" pod "369aea3b-19ba-40a9-a941-884322eb7f51" (UID: "369aea3b-19ba-40a9-a941-884322eb7f51"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:31.595472 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.595401    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-cacerts" pod "369aea3b-19ba-40a9-a941-884322eb7f51" (UID: "369aea3b-19ba-40a9-a941-884322eb7f51"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:31.595653 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.595625    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-kube-api-access-z7jf5" pod "369aea3b-19ba-40a9-a941-884322eb7f51" (UID: "369aea3b-19ba-40a9-a941-884322eb7f51"). InnerVolumeSpecName "kube-api-access-z7jf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:31.595743 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.595717    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-dns-cert" pod "369aea3b-19ba-40a9-a941-884322eb7f51" (UID: "369aea3b-19ba-40a9-a941-884322eb7f51"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:31.693777 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.693719    4853 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-kubeconfig\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:31.693777 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.693747    4853 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/369aea3b-19ba-40a9-a941-884322eb7f51-local-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:31.693777 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.693760    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7jf5\" (UniqueName: \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-kube-api-access-z7jf5\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:31.693777 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.693768    4853 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-dns-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:31.693777 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.693776    4853 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/369aea3b-19ba-40a9-a941-884322eb7f51-cacerts\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:31.694005 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.693786    4853 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/369aea3b-19ba-40a9-a941-884322eb7f51-istio-csr-ca-configmap\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:31.694005 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:31.693795    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/369aea3b-19ba-40a9-a941-884322eb7f51-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:32.074854 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:32.074825    4853 generic.go:356] "Generic (PLEG): container finished" podID="369aea3b-19ba-40a9-a941-884322eb7f51" containerID="d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338" exitCode=0
Oct 30 18:47:32.075247 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:32.074879    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"
Oct 30 18:47:32.075247 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:32.074913    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5" event={"ID":"369aea3b-19ba-40a9-a941-884322eb7f51","Type":"ContainerDied","Data":"d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338"}
Oct 30 18:47:32.075247 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:32.074964    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5" event={"ID":"369aea3b-19ba-40a9-a941-884322eb7f51","Type":"ContainerDied","Data":"e6a013a287510a10c06d243122c08860e5444932cf983d28d5b56ad20afe429b"}
Oct 30 18:47:32.075247 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:32.074987    4853 scope.go:122] "RemoveContainer" containerID="d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338"
Oct 30 18:47:32.083808 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:32.083774    4853 scope.go:122] "RemoveContainer" containerID="d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338"
Oct 30 18:47:32.084106 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:47:32.084086    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338\": container with ID starting with d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338 not found: ID does not exist" containerID="d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338"
Oct 30 18:47:32.084178 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:32.084115    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338"} err="failed to get container status \"d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338\": rpc error: code = NotFound desc = could not find container \"d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338\": container with ID starting with d26beab1a7245baa4e59534dd31476125788ee6392c6100df6439cab3ab38338 not found: ID does not exist"
Oct 30 18:47:32.232877 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:32.232826    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"]
Oct 30 18:47:32.278928 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:32.278905    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-j4sp5"]
Oct 30 18:47:33.939188 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:33.939151    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369aea3b-19ba-40a9-a941-884322eb7f51" path="/var/lib/kubelet/pods/369aea3b-19ba-40a9-a941-884322eb7f51/volumes"
Oct 30 18:47:36.058161 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:36.058126    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:42.793970 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:42.793939    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:42.884547 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:42.884509    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"]
Oct 30 18:47:42.884818 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:42.884805    4853 cpu_manager.go:414] "containerMap: removing container" podUID="369aea3b-19ba-40a9-a941-884322eb7f51" containerName="discovery"
Oct 30 18:47:42.884860 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:42.884821    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369aea3b-19ba-40a9-a941-884322eb7f51" containerName="discovery"
Oct 30 18:47:42.884903 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:42.884894    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="369aea3b-19ba-40a9-a941-884322eb7f51" containerName="discovery"
Oct 30 18:47:42.888864 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:42.888849    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"
Oct 30 18:47:42.941932 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:42.941910    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"]
Oct 30 18:47:42.970485 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:42.970463    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6755790-8a5d-4e49-9ba3-5f846cde1b25-cert\") pod \"llmisvc-controller-manager-67f87c4bbb-szjsd\" (UID: \"b6755790-8a5d-4e49-9ba3-5f846cde1b25\") " pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"
Oct 30 18:47:42.970645 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:42.970507    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szjgv\" (UniqueName: \"kubernetes.io/projected/b6755790-8a5d-4e49-9ba3-5f846cde1b25-kube-api-access-szjgv\") pod \"llmisvc-controller-manager-67f87c4bbb-szjsd\" (UID: \"b6755790-8a5d-4e49-9ba3-5f846cde1b25\") " pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"
Oct 30 18:47:42.986473 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:42.986454    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-2kvh8\""
Oct 30 18:47:42.986589 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:42.986577    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Oct 30 18:47:43.018676 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:43.018655    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:43.033026 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:43.032998    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 18:47:43.033026 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:43.033001    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 18:47:43.071302 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:43.071250    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6755790-8a5d-4e49-9ba3-5f846cde1b25-cert\") pod \"llmisvc-controller-manager-67f87c4bbb-szjsd\" (UID: \"b6755790-8a5d-4e49-9ba3-5f846cde1b25\") " pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"
Oct 30 18:47:43.071400 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:43.071297    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szjgv\" (UniqueName: \"kubernetes.io/projected/b6755790-8a5d-4e49-9ba3-5f846cde1b25-kube-api-access-szjgv\") pod \"llmisvc-controller-manager-67f87c4bbb-szjsd\" (UID: \"b6755790-8a5d-4e49-9ba3-5f846cde1b25\") " pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"
Oct 30 18:47:43.073857 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:43.073839    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6755790-8a5d-4e49-9ba3-5f846cde1b25-cert\") pod \"llmisvc-controller-manager-67f87c4bbb-szjsd\" (UID: \"b6755790-8a5d-4e49-9ba3-5f846cde1b25\") " pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"
Oct 30 18:47:43.153112 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:43.153090    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szjgv\" (UniqueName: \"kubernetes.io/projected/b6755790-8a5d-4e49-9ba3-5f846cde1b25-kube-api-access-szjgv\") pod \"llmisvc-controller-manager-67f87c4bbb-szjsd\" (UID: \"b6755790-8a5d-4e49-9ba3-5f846cde1b25\") " pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"
Oct 30 18:47:43.200658 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:43.200636    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"
Oct 30 18:47:43.307506 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:43.307478    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:43.566393 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:43.566365    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"]
Oct 30 18:47:43.572851 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:47:43.572822    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6755790_8a5d_4e49_9ba3_5f846cde1b25.slice/crio-eccf57437663875a8a29433cbef634a5e368d3748c413ffd1306e5520333deab WatchSource:0}: Error finding container eccf57437663875a8a29433cbef634a5e368d3748c413ffd1306e5520333deab: Status 404 returned error can't find the container with id eccf57437663875a8a29433cbef634a5e368d3748c413ffd1306e5520333deab
Oct 30 18:47:43.574403 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:43.574380    4853 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:47:44.135189 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:44.135150    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd" event={"ID":"b6755790-8a5d-4e49-9ba3-5f846cde1b25","Type":"ContainerStarted","Data":"eccf57437663875a8a29433cbef634a5e368d3748c413ffd1306e5520333deab"}
Oct 30 18:47:44.354389 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:44.354346    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:44.579562 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:44.579512    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:47.152095 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:47.152068    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd" event={"ID":"b6755790-8a5d-4e49-9ba3-5f846cde1b25","Type":"ContainerStarted","Data":"f1470471460c07904e5270034d291bf7c9cb241abbe2bda4b3e158b965accd96"}
Oct 30 18:47:47.152448 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:47.152129    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"
Oct 30 18:47:47.296342 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:47.296260    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd" podStartSLOduration=1.8403251790000001 podStartE2EDuration="5.296246409s" totalImagesPullingTime="3.45592123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:47.294808801 +0000 UTC m=+573.465165650" watchObservedRunningTime="2026-10-30 18:47:47.296246409 +0000 UTC m=+573.466603257"
Oct 30 18:47:49.874493 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:47:49.874450    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:47:51.677658 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:47:51.677628    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:17.343201 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:48:17.343163    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:18.158229 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:48:18.158193    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67f87c4bbb-szjsd"
Oct 30 18:48:19.873248 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:48:19.873209    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:48:39.940483 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:48:39.940432    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:40.233259 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:48:40.233182    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:49.879690 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:48:49.879639    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:48:52.119511 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:48:52.119464    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:53.862656 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:48:53.862617    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:08.407868 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.407839    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:08.532632 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.532604    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-4kw9s"]
Oct 30 18:49:08.535941 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.535922    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:08.562734 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.562712    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d131e3-6b96-4673-b1af-37fc5c721c44-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-4kw9s\" (UID: \"35d131e3-6b96-4673-b1af-37fc5c721c44\") " pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:08.562832 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.562746    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35d131e3-6b96-4673-b1af-37fc5c721c44-cert\") pod \"odh-model-controller-7f9f8cdc65-4kw9s\" (UID: \"35d131e3-6b96-4673-b1af-37fc5c721c44\") " pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:08.562832 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.562762    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxsrh\" (UniqueName: \"kubernetes.io/projected/35d131e3-6b96-4673-b1af-37fc5c721c44-kube-api-access-rxsrh\") pod \"odh-model-controller-7f9f8cdc65-4kw9s\" (UID: \"35d131e3-6b96-4673-b1af-37fc5c721c44\") " pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:08.629908 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.629885    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-4kw9s"]
Oct 30 18:49:08.635244 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.635223    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-7zxbj\""
Oct 30 18:49:08.635365 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.635350    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Oct 30 18:49:08.635485 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.635352    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Oct 30 18:49:08.636007 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.635991    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:08.663993 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.663915    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d131e3-6b96-4673-b1af-37fc5c721c44-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-4kw9s\" (UID: \"35d131e3-6b96-4673-b1af-37fc5c721c44\") " pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:08.663993 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.663956    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35d131e3-6b96-4673-b1af-37fc5c721c44-cert\") pod \"odh-model-controller-7f9f8cdc65-4kw9s\" (UID: \"35d131e3-6b96-4673-b1af-37fc5c721c44\") " pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:08.663993 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.663984    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxsrh\" (UniqueName: \"kubernetes.io/projected/35d131e3-6b96-4673-b1af-37fc5c721c44-kube-api-access-rxsrh\") pod \"odh-model-controller-7f9f8cdc65-4kw9s\" (UID: \"35d131e3-6b96-4673-b1af-37fc5c721c44\") " pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:08.666583 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.666558    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35d131e3-6b96-4673-b1af-37fc5c721c44-cert\") pod \"odh-model-controller-7f9f8cdc65-4kw9s\" (UID: \"35d131e3-6b96-4673-b1af-37fc5c721c44\") " pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:08.666671 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.666604    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d131e3-6b96-4673-b1af-37fc5c721c44-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-4kw9s\" (UID: \"35d131e3-6b96-4673-b1af-37fc5c721c44\") " pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:08.725192 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.725170    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxsrh\" (UniqueName: \"kubernetes.io/projected/35d131e3-6b96-4673-b1af-37fc5c721c44-kube-api-access-rxsrh\") pod \"odh-model-controller-7f9f8cdc65-4kw9s\" (UID: \"35d131e3-6b96-4673-b1af-37fc5c721c44\") " pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:08.848515 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.848488    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:08.860336 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:08.860296    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:09.218411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:09.218380    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-4kw9s"]
Oct 30 18:49:09.225199 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:49:09.225173    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d131e3_6b96_4673_b1af_37fc5c721c44.slice/crio-0b906927718000e690a14cbfedde2fc4aea9717336be69e81ca7246f2e28642d WatchSource:0}: Error finding container 0b906927718000e690a14cbfedde2fc4aea9717336be69e81ca7246f2e28642d: Status 404 returned error can't find the container with id 0b906927718000e690a14cbfedde2fc4aea9717336be69e81ca7246f2e28642d
Oct 30 18:49:09.521551 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:09.521454    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s" event={"ID":"35d131e3-6b96-4673-b1af-37fc5c721c44","Type":"ContainerStarted","Data":"0b906927718000e690a14cbfedde2fc4aea9717336be69e81ca7246f2e28642d"}
Oct 30 18:49:13.542717 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:13.542688    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s" event={"ID":"35d131e3-6b96-4673-b1af-37fc5c721c44","Type":"ContainerStarted","Data":"2f60c7719e1320c7e259c866b98bb22ab0acba1384387051778d5eb5a54ef79a"}
Oct 30 18:49:13.543154 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:13.542809    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:13.653220 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:13.653172    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s" podStartSLOduration=1.865859956 podStartE2EDuration="5.653158345s" totalImagesPullingTime="3.787298389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:13.651424926 +0000 UTC m=+659.821781773" watchObservedRunningTime="2026-10-30 18:49:13.653158345 +0000 UTC m=+659.823515192"
Oct 30 18:49:14.046729 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:14.046704    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:14.276078 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:14.276048    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:15.057879 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:15.057844    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:19.875034 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:49:19.874987    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:49:24.548627 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:24.548599    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7f9f8cdc65-4kw9s"
Oct 30 18:49:49.874719 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:49:49.874675    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:49:52.330310 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:49:52.330261    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:19.876079 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:50:19.876039    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:50:46.419245 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:50:46.419204    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:46.688826 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:50:46.688744    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:49.876270 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:50:49.876229    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:50:52.651905 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:50:52.651877    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:56.718570 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:50:56.718512    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:10.214979 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:10.214949    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:10.433394 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:10.433354    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:19.873049 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:51:19.873011    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:51:20.234379 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:20.234302    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:20.438738 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:20.438704    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:26.532549 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:26.532497    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:28.151480 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:28.151442    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:29.517746 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.517713    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"]
Oct 30 18:51:29.521003 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.520981    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.575979 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.575955    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Oct 30 18:51:29.577359 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.577334    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"]
Oct 30 18:51:29.622282 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.622261    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Oct 30 18:51:29.622440 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.622419    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-hqvzl\""
Oct 30 18:51:29.622583 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.622565    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Oct 30 18:51:29.644747 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.644722    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.644861 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.644758    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.644913 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.644857    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87zl\" (UniqueName: \"kubernetes.io/projected/accbe539-0367-45e5-b4b3-7b3efffd87c6-kube-api-access-x87zl\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.644951 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.644926    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accbe539-0367-45e5-b4b3-7b3efffd87c6-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.644989 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.644961    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.644989 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.644978    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.745764 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.745736    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87zl\" (UniqueName: \"kubernetes.io/projected/accbe539-0367-45e5-b4b3-7b3efffd87c6-kube-api-access-x87zl\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.745892 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.745769    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accbe539-0367-45e5-b4b3-7b3efffd87c6-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.745892 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.745804    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.745892 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.745821    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.746081 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.745927    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.746081 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.745967    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.746220 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.746199    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.746279 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.746218    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.746345 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.746327    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.748310 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.748288    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.748575 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.748558    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accbe539-0367-45e5-b4b3-7b3efffd87c6-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.770380 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.770327    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:29.801090 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.801065    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87zl\" (UniqueName: \"kubernetes.io/projected/accbe539-0367-45e5-b4b3-7b3efffd87c6-kube-api-access-x87zl\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-m49kb\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:29.835450 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:29.835433    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:30.203677 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:30.203649    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"]
Oct 30 18:51:30.210135 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:51:30.210111    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccbe539_0367_45e5_b4b3_7b3efffd87c6.slice/crio-74028aee50960c38cef8cae862dc530d325faeb20ce49c0740e790c78377d3f8 WatchSource:0}: Error finding container 74028aee50960c38cef8cae862dc530d325faeb20ce49c0740e790c78377d3f8: Status 404 returned error can't find the container with id 74028aee50960c38cef8cae862dc530d325faeb20ce49c0740e790c78377d3f8
Oct 30 18:51:30.285147 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:30.285062    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:31.167322 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:31.167284    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb" event={"ID":"accbe539-0367-45e5-b4b3-7b3efffd87c6","Type":"ContainerStarted","Data":"74028aee50960c38cef8cae862dc530d325faeb20ce49c0740e790c78377d3f8"}
Oct 30 18:51:34.182642 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:34.182569    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb" event={"ID":"accbe539-0367-45e5-b4b3-7b3efffd87c6","Type":"ContainerStarted","Data":"1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1"}
Oct 30 18:51:34.313469 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:34.313416    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb" podStartSLOduration=2.3781696070000002 podStartE2EDuration="5.313401137s" totalImagesPullingTime="2.93523153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:34.311471311 +0000 UTC m=+800.481828180" watchObservedRunningTime="2026-10-30 18:51:34.313401137 +0000 UTC m=+800.483757997"
Oct 30 18:51:34.839554 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:34.836616    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:36.785885 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:36.785849    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:37.015497 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:37.015456    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:37.368160 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:37.368085    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:39.836097 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:39.836061    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:39.855046 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:39.855022    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:40.219589 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:40.219504    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:51:43.225558 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:43.224752    4853 generic.go:356] "Generic (PLEG): container finished" podID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerID="dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356" exitCode=0
Oct 30 18:51:43.225558 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:43.224898    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" event={"ID":"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9","Type":"ContainerDied","Data":"dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356"}
Oct 30 18:51:43.225558 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:43.225367    4853 scope.go:122] "RemoveContainer" containerID="dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356"
Oct 30 18:51:43.227172 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:43.226953    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:43.227172 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:43.227123    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:43.227346 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:43.227238    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:43.277705 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:43.277664    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:44.228676 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:44.228646    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:44.230737 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:44.230711    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" event={"ID":"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9","Type":"ContainerStarted","Data":"8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832"}
Oct 30 18:51:44.463555 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:44.462128    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:44.741893 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:44.741864    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:51:44.742243 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:44.742220    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Oct 30 18:51:44.742396 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:44.742371    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Oct 30 18:51:45.750552 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:45.749865    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Oct 30 18:51:45.750552 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:45.749924    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Oct 30 18:51:46.740896 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:46.740859    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Oct 30 18:51:46.741177 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:46.741149    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Oct 30 18:51:47.744557 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:47.741162    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Oct 30 18:51:47.744557 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:47.741227    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Oct 30 18:51:47.974314 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:47.974283    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"]
Oct 30 18:51:47.981942 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:47.981917    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.081137 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.081110    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"]
Oct 30 18:51:48.097624 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.097597    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.097759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.097634    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.097759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.097654    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.097759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.097732    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.097866 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.097836    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.097899 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.097873    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.097943 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.097902    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.098019 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.097997    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4pz\" (UniqueName: \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-kube-api-access-mv4pz\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.098074 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.098057    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.199370 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199340    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.199370 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199375    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.199602 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199391    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.199602 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199409    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4pz\" (UniqueName: \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-kube-api-access-mv4pz\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.199602 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199431    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.199602 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199447    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.199602 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199469    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.199602 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199496    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.199602 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199566    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.199934 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199825    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.199934 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199917    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.200051 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.199978    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.200137 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.200113    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.200224 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.200207    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.201942 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.201916    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.202086 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.202071    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.261218 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.261165    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.266497 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.266468    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4pz\" (UniqueName: \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-kube-api-access-mv4pz\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bjdf6\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.294662 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.294638    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:48.742603 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.738827    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body=
Oct 30 18:51:48.742603 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.738884    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused"
Oct 30 18:51:48.760435 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.760406    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"]
Oct 30 18:51:48.767622 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:51:48.767598    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079a6eea_fc68_40fd_a4ac_327ac73abe2d.slice/crio-1d221182de67e6cb7dc282a123ee90933c71e1887a99299520e81e76d23c5fea WatchSource:0}: Error finding container 1d221182de67e6cb7dc282a123ee90933c71e1887a99299520e81e76d23c5fea: Status 404 returned error can't find the container with id 1d221182de67e6cb7dc282a123ee90933c71e1887a99299520e81e76d23c5fea
Oct 30 18:51:48.769891 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.769861    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:48.769950 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.769924    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:48.769986 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:48.769957    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:51:49.276717 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:49.276683    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" event={"ID":"079a6eea-fc68-40fd-a4ac-327ac73abe2d","Type":"ContainerStarted","Data":"d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62"}
Oct 30 18:51:49.276897 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:49.276738    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" event={"ID":"079a6eea-fc68-40fd-a4ac-327ac73abe2d","Type":"ContainerStarted","Data":"1d221182de67e6cb7dc282a123ee90933c71e1887a99299520e81e76d23c5fea"}
Oct 30 18:51:49.294784 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:49.294762    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:49.296192 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:49.296172    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 18:51:49.296263 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:49.296214    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 18:51:49.390888 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:49.390832    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" podStartSLOduration=2.390818229 podStartE2EDuration="2.390818229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:49.389512591 +0000 UTC m=+815.559869441" watchObservedRunningTime="2026-10-30 18:51:49.390818229 +0000 UTC m=+815.561175149"
Oct 30 18:51:49.763556 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:49.762888    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:51:49.764138 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:49.764117    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:51:49.764867 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:49.764847    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 18:51:49.889693 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:51:49.889642    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:51:50.295246 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:50.295141    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 18:51:50.295246 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:50.295189    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 18:51:51.295624 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:51.295591    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 18:51:51.296091 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:51.295653    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 18:51:52.295236 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:52.295205    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 18:51:52.295406 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:52.295265    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 18:51:52.983411 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:52.983381    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:53.295292 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:53.295217    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 18:51:53.295292 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:53.295263    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 18:51:54.295331 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:54.295295    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 30 18:51:54.295773 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:54.295350    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 30 18:51:55.299164 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:55.299136    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:55.299688 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:55.299509    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:55.300190 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:55.300171    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 18:51:57.942900 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:51:57.942861    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:11.406419 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:11.406388    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"]
Oct 30 18:52:11.415348 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:52:11.415311    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1" cmd=["/bin/sleep","15"]
Oct 30 18:52:11.415506 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:11.415367    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 18:52:11.415506 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:52:11.415379    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb" podUID="accbe539-0367-45e5-b4b3-7b3efffd87c6" containerName="main" containerID="cri-o://1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1"
Oct 30 18:52:11.415506 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:11.415425    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb" podUID="accbe539-0367-45e5-b4b3-7b3efffd87c6" containerName="main" containerID="cri-o://1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1" gracePeriod=60
Oct 30 18:52:11.480882 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:11.480856    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:11.689077 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:11.688994    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:12.158046 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.158025    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:52:12.305975 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.305910    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/accbe539-0367-45e5-b4b3-7b3efffd87c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/accbe539-0367-45e5-b4b3-7b3efffd87c6-tls-certs\") pod \"accbe539-0367-45e5-b4b3-7b3efffd87c6\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") "
Oct 30 18:52:12.305975 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.305943    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-dshm\") pod \"accbe539-0367-45e5-b4b3-7b3efffd87c6\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") "
Oct 30 18:52:12.305975 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.305967    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-tmp-dir\") pod \"accbe539-0367-45e5-b4b3-7b3efffd87c6\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") "
Oct 30 18:52:12.306215 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.305992    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-home\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-home\") pod \"accbe539-0367-45e5-b4b3-7b3efffd87c6\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") "
Oct 30 18:52:12.306215 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.306052    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-model-cache\") pod \"accbe539-0367-45e5-b4b3-7b3efffd87c6\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") "
Oct 30 18:52:12.306215 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.306087    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/accbe539-0367-45e5-b4b3-7b3efffd87c6-kube-api-access-x87zl\" (UniqueName: \"kubernetes.io/projected/accbe539-0367-45e5-b4b3-7b3efffd87c6-kube-api-access-x87zl\") pod \"accbe539-0367-45e5-b4b3-7b3efffd87c6\" (UID: \"accbe539-0367-45e5-b4b3-7b3efffd87c6\") "
Oct 30 18:52:12.306378 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.306255    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-tmp-dir" pod "accbe539-0367-45e5-b4b3-7b3efffd87c6" (UID: "accbe539-0367-45e5-b4b3-7b3efffd87c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:12.306378 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.306298    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-home" pod "accbe539-0367-45e5-b4b3-7b3efffd87c6" (UID: "accbe539-0367-45e5-b4b3-7b3efffd87c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:12.306378 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.306361    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:12.306378 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.306358    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-model-cache" pod "accbe539-0367-45e5-b4b3-7b3efffd87c6" (UID: "accbe539-0367-45e5-b4b3-7b3efffd87c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:12.308329 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.308301    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/accbe539-0367-45e5-b4b3-7b3efffd87c6-tls-certs" pod "accbe539-0367-45e5-b4b3-7b3efffd87c6" (UID: "accbe539-0367-45e5-b4b3-7b3efffd87c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:12.308329 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.308322    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-dshm" pod "accbe539-0367-45e5-b4b3-7b3efffd87c6" (UID: "accbe539-0367-45e5-b4b3-7b3efffd87c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:52:12.308471 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.308326    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accbe539-0367-45e5-b4b3-7b3efffd87c6-kube-api-access-x87zl" pod "accbe539-0367-45e5-b4b3-7b3efffd87c6" (UID: "accbe539-0367-45e5-b4b3-7b3efffd87c6"). InnerVolumeSpecName "kube-api-access-x87zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:12.382933 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.382910    4853 generic.go:356] "Generic (PLEG): container finished" podID="accbe539-0367-45e5-b4b3-7b3efffd87c6" containerID="1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1" exitCode=0
Oct 30 18:52:12.383047 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.382952    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"
Oct 30 18:52:12.383047 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.383008    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb" event={"ID":"accbe539-0367-45e5-b4b3-7b3efffd87c6","Type":"ContainerDied","Data":"1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1"}
Oct 30 18:52:12.383144 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.383063    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb" event={"ID":"accbe539-0367-45e5-b4b3-7b3efffd87c6","Type":"ContainerDied","Data":"74028aee50960c38cef8cae862dc530d325faeb20ce49c0740e790c78377d3f8"}
Oct 30 18:52:12.383144 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.383093    4853 scope.go:122] "RemoveContainer" containerID="1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1"
Oct 30 18:52:12.391937 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.391922    4853 scope.go:122] "RemoveContainer" containerID="1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1"
Oct 30 18:52:12.392185 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:52:12.392168    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1\": container with ID starting with 1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1 not found: ID does not exist" containerID="1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1"
Oct 30 18:52:12.392227 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.392191    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1"} err="failed to get container status \"1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1\": rpc error: code = NotFound desc = could not find container \"1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1\": container with ID starting with 1bd9ab1a289e39dc549e3bddef3e8dd9cb63a77fe7e15f42a0cd13ebfaa763d1 not found: ID does not exist"
Oct 30 18:52:12.407542 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.407514    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accbe539-0367-45e5-b4b3-7b3efffd87c6-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:12.407805 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.407558    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:12.407805 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.407567    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:12.407805 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.407574    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accbe539-0367-45e5-b4b3-7b3efffd87c6-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:12.407805 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.407585    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x87zl\" (UniqueName: \"kubernetes.io/projected/accbe539-0367-45e5-b4b3-7b3efffd87c6-kube-api-access-x87zl\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:12.539722 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.539697    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"]
Oct 30 18:52:12.544806 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:12.544788    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-m49kb"]
Oct 30 18:52:13.938109 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:13.938078    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="accbe539-0367-45e5-b4b3-7b3efffd87c6" path="/var/lib/kubelet/pods/accbe539-0367-45e5-b4b3-7b3efffd87c6/volumes"
Oct 30 18:52:19.876021 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:52:19.875975    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:52:49.875718 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:52:49.875674    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:52:53.317324 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:53.317278    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:58.761694 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:58.761661    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:59.062646 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:52:59.062563    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:06.334804 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.334752    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:06.353960 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.353927    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"]
Oct 30 18:53:06.354310 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.354293    4853 cpu_manager.go:414] "containerMap: removing container" podUID="accbe539-0367-45e5-b4b3-7b3efffd87c6" containerName="main"
Oct 30 18:53:06.354397 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.354312    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="accbe539-0367-45e5-b4b3-7b3efffd87c6" containerName="main"
Oct 30 18:53:06.354397 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.354385    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="accbe539-0367-45e5-b4b3-7b3efffd87c6" containerName="main"
Oct 30 18:53:06.357350 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.357329    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.400330 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.400304    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m477z\" (UniqueName: \"kubernetes.io/projected/a44120e6-3289-41db-b966-850b6f54431f-kube-api-access-m477z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.400448 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.400355    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.400491 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.400451    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.400556 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.400501    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.400639 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.400622    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44120e6-3289-41db-b966-850b6f54431f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.400696 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.400678    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.410016 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.409995    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Oct 30 18:53:06.410016 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.410001    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Oct 30 18:53:06.456418 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.456393    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Oct 30 18:53:06.456418 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.456411    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-mx5ss\""
Oct 30 18:53:06.458562 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.458520    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"]
Oct 30 18:53:06.501294 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.501268    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m477z\" (UniqueName: \"kubernetes.io/projected/a44120e6-3289-41db-b966-850b6f54431f-kube-api-access-m477z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.501393 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.501307    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.501393 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.501344    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.501513 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.501491    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.501600 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.501571    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44120e6-3289-41db-b966-850b6f54431f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.501654 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.501613    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.501806 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.501786    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.501892 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.501873    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.501942 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.501906    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.503882 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.503862    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.504118 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.504099    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44120e6-3289-41db-b966-850b6f54431f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.555064 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.555042    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:06.556317 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.556299    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m477z\" (UniqueName: \"kubernetes.io/projected/a44120e6-3289-41db-b966-850b6f54431f-kube-api-access-m477z\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:06.670454 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:06.670382    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:07.010516 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:07.010429    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:07.044826 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:07.044799    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"]
Oct 30 18:53:07.051945 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:53:07.051912    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44120e6_3289_41db_b966_850b6f54431f.slice/crio-5c1cbb94086aa565d7269581ef6a45ffbc4da48378f224a54ebfd50f140e6e35 WatchSource:0}: Error finding container 5c1cbb94086aa565d7269581ef6a45ffbc4da48378f224a54ebfd50f140e6e35: Status 404 returned error can't find the container with id 5c1cbb94086aa565d7269581ef6a45ffbc4da48378f224a54ebfd50f140e6e35
Oct 30 18:53:07.646584 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:07.646554    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq" event={"ID":"a44120e6-3289-41db-b966-850b6f54431f","Type":"ContainerStarted","Data":"3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4"}
Oct 30 18:53:07.646967 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:07.646605    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq" event={"ID":"a44120e6-3289-41db-b966-850b6f54431f","Type":"ContainerStarted","Data":"5c1cbb94086aa565d7269581ef6a45ffbc4da48378f224a54ebfd50f140e6e35"}
Oct 30 18:53:07.670566 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:07.670520    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:07.758056 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:07.758007    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq" podStartSLOduration=1.7579939009999999 podStartE2EDuration="1.757993901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:07.756432364 +0000 UTC m=+893.926789214" watchObservedRunningTime="2026-10-30 18:53:07.757993901 +0000 UTC m=+893.928350749"
Oct 30 18:53:13.202076 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:13.202048    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:13.423463 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:13.423433    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:16.671423 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:16.671388    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:16.687565 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:16.687520    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:16.719044 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:16.719020    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:19.878268 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:53:19.878233    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:53:20.196636 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:20.196504    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:23.604152 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:23.604116    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:23.817777 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:23.817743    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:28.101985 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:28.101949    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:28.299187 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:28.299160    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:30.102095 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:30.102059    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:30.311496 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:30.311466    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:30.581283 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:30.581247    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:33.159727 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.159697    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"]
Oct 30 18:53:33.162477 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.162458    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.206309 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.206289    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"]
Oct 30 18:53:33.209712 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.209696    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.255053 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.255028    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"]
Oct 30 18:53:33.303504 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.303473    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"]
Oct 30 18:53:33.344280 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344251    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.344413 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344292    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.344413 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344316    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.344413 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344375    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.344600 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344446    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjz8\" (UniqueName: \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-kube-api-access-gjjz8\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.344600 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344500    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.344600 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344516    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.344600 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344555    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.344600 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344578    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.344846 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344653    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.344846 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344687    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7d6t\" (UniqueName: \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-kube-api-access-v7d6t\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.344846 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344722    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.344846 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344772    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724ac112-9245-4ae5-aa46-ddc1ea352431-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.344846 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344814    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.344846 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344844    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.345139 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344862    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.345139 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344880    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.345139 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.344898    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.446256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446194    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.446256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446224    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724ac112-9245-4ae5-aa46-ddc1ea352431-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.446256 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446244    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.446484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446263    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.446484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446286    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.446484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446311    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.446484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446333    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.446484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446351    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.446484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446389    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.446484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446411    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.446484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446443    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.446484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446469    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjz8\" (UniqueName: \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-kube-api-access-gjjz8\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.447021 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446496    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.447021 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446520    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.447021 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446571    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.447021 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446598    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.447021 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446629    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.447021 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446668    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7d6t\" (UniqueName: \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-kube-api-access-v7d6t\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.447021 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446766    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.447021 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.446857    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.447443 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.447073    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724ac112-9245-4ae5-aa46-ddc1ea352431-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.447443 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.447103    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.447443 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.447148    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.447443 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.447194    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.447443 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.447263    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.447729 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.447556    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.447833 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.447807    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.447904 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.447790    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.449274 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.449256    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.449883 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.449858    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.449985 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.449945    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.449985 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.449959    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.501776 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.501746    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.501887 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.501864    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7d6t\" (UniqueName: \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-kube-api-access-v7d6t\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.501992 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.501972    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjz8\" (UniqueName: \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-kube-api-access-gjjz8\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2lqh4\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.502160 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.502143    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-6c8rk\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.522100 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.522081    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:33.774730 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.774605    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:33.955415 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.955385    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"]
Oct 30 18:53:33.962192 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:53:33.962162    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724ac112_9245_4ae5_aa46_ddc1ea352431.slice/crio-ab1f51e37344c8b6f4b35514c721084069fb45d02fe4b907326eebaeea4b498e WatchSource:0}: Error finding container ab1f51e37344c8b6f4b35514c721084069fb45d02fe4b907326eebaeea4b498e: Status 404 returned error can't find the container with id ab1f51e37344c8b6f4b35514c721084069fb45d02fe4b907326eebaeea4b498e
Oct 30 18:53:33.964156 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.964125    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:33.964237 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.964187    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:33.964237 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:33.964216    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:34.200969 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:34.200940    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"]
Oct 30 18:53:34.207489 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:53:34.207462    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407e5a69_464d_4c90_8ab0_2416b9bfe8c8.slice/crio-3fcbc32ddd70cc993cf205690a74df6bd6ac5218464cfe5ff2ef22d570f87bd1 WatchSource:0}: Error finding container 3fcbc32ddd70cc993cf205690a74df6bd6ac5218464cfe5ff2ef22d570f87bd1: Status 404 returned error can't find the container with id 3fcbc32ddd70cc993cf205690a74df6bd6ac5218464cfe5ff2ef22d570f87bd1
Oct 30 18:53:34.209729 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:34.209691    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:34.209806 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:34.209773    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:34.209843 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:34.209820    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 18:53:34.798266 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:34.798233    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" event={"ID":"407e5a69-464d-4c90-8ab0-2416b9bfe8c8","Type":"ContainerStarted","Data":"f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1"}
Oct 30 18:53:34.798467 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:34.798283    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" event={"ID":"407e5a69-464d-4c90-8ab0-2416b9bfe8c8","Type":"ContainerStarted","Data":"3fcbc32ddd70cc993cf205690a74df6bd6ac5218464cfe5ff2ef22d570f87bd1"}
Oct 30 18:53:34.799926 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:34.799903    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" event={"ID":"724ac112-9245-4ae5-aa46-ddc1ea352431","Type":"ContainerStarted","Data":"a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812"}
Oct 30 18:53:34.800052 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:34.799947    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" event={"ID":"724ac112-9245-4ae5-aa46-ddc1ea352431","Type":"ContainerStarted","Data":"ab1f51e37344c8b6f4b35514c721084069fb45d02fe4b907326eebaeea4b498e"}
Oct 30 18:53:34.924758 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:34.924707    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" podStartSLOduration=1.924690537 podStartE2EDuration="1.924690537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:34.922813068 +0000 UTC m=+921.093169916" watchObservedRunningTime="2026-10-30 18:53:34.924690537 +0000 UTC m=+921.095047384"
Oct 30 18:53:35.523119 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:35.523089    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:35.524328 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:35.524303    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:53:35.524428 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:35.524343    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" podUID="724ac112-9245-4ae5-aa46-ddc1ea352431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 18:53:35.776938 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:35.776850    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:35.785566 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:35.785139    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:53:35.785566 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:35.785201    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" podUID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 30 18:53:36.522920 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:36.522888    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:53:36.523135 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:36.522939    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" podUID="724ac112-9245-4ae5-aa46-ddc1ea352431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 18:53:36.779612 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:36.778899    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:53:36.779612 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:36.778962    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" podUID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 30 18:53:37.522458 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:37.522414    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:53:37.522648 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:37.522479    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" podUID="724ac112-9245-4ae5-aa46-ddc1ea352431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 18:53:37.661364 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:37.661329    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:37.779613 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:37.778956    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:53:37.779613 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:37.779040    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" podUID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 30 18:53:38.523313 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:38.523281    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:53:38.523654 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:38.523631    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" podUID="724ac112-9245-4ae5-aa46-ddc1ea352431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 18:53:38.675490 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:38.675454    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:38.776961 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:38.776873    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:53:38.777237 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:38.777209    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" podUID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 30 18:53:38.891556 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:38.891502    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:39.522934 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:39.522892    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 30 18:53:39.523193 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:39.523171    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" podUID="724ac112-9245-4ae5-aa46-ddc1ea352431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 30 18:53:39.778991 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:39.778888    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 30 18:53:39.779249 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:39.779218    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" podUID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 30 18:53:40.526919 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:40.526889    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:40.527355 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:40.527251    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:40.528060 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:40.528039    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 18:53:40.688602 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:40.688560    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" podStartSLOduration=7.688547558 podStartE2EDuration="7.688547558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:35.056393735 +0000 UTC m=+921.226750584" watchObservedRunningTime="2026-10-30 18:53:40.688547558 +0000 UTC m=+926.858904402"
Oct 30 18:53:40.778596 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:40.778521    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:40.778825 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:40.778807    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:40.779589 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:40.779569    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 18:53:45.577554 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:45.574072    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"]
Oct 30 18:53:45.592559 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:45.592098    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:45.663198 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:53:45.663138    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4" cmd=["/bin/sleep","15"]
Oct 30 18:53:45.663575 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:45.663548    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 18:53:45.663727 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:53:45.663705    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq" podUID="a44120e6-3289-41db-b966-850b6f54431f" containerName="main" containerID="cri-o://3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4"
Oct 30 18:53:45.663879 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:45.663859    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq" podUID="a44120e6-3289-41db-b966-850b6f54431f" containerName="main" containerID="cri-o://3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4" gracePeriod=60
Oct 30 18:53:45.708765 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:45.708722    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq" podUID="a44120e6-3289-41db-b966-850b6f54431f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Oct 30 18:53:45.834391 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:45.834331    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:46.026387 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.026366    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:46.116081 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116013    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-tmp-dir\") pod \"a44120e6-3289-41db-b966-850b6f54431f\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") "
Oct 30 18:53:46.116081 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116076    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-home\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-home\") pod \"a44120e6-3289-41db-b966-850b6f54431f\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") "
Oct 30 18:53:46.116278 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116110    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-model-cache\") pod \"a44120e6-3289-41db-b966-850b6f54431f\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") "
Oct 30 18:53:46.116278 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116150    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44120e6-3289-41db-b966-850b6f54431f-tls-certs\" (UniqueName: \"kubernetes.io/secret/a44120e6-3289-41db-b966-850b6f54431f-tls-certs\") pod \"a44120e6-3289-41db-b966-850b6f54431f\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") "
Oct 30 18:53:46.116278 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116218    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-dshm\") pod \"a44120e6-3289-41db-b966-850b6f54431f\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") "
Oct 30 18:53:46.116278 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116261    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44120e6-3289-41db-b966-850b6f54431f-kube-api-access-m477z\" (UniqueName: \"kubernetes.io/projected/a44120e6-3289-41db-b966-850b6f54431f-kube-api-access-m477z\") pod \"a44120e6-3289-41db-b966-850b6f54431f\" (UID: \"a44120e6-3289-41db-b966-850b6f54431f\") "
Oct 30 18:53:46.116467 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116275    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-tmp-dir" pod "a44120e6-3289-41db-b966-850b6f54431f" (UID: "a44120e6-3289-41db-b966-850b6f54431f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:46.116467 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116344    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-home" pod "a44120e6-3289-41db-b966-850b6f54431f" (UID: "a44120e6-3289-41db-b966-850b6f54431f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:46.116467 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116409    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-model-cache" pod "a44120e6-3289-41db-b966-850b6f54431f" (UID: "a44120e6-3289-41db-b966-850b6f54431f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:46.116467 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116462    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:46.116626 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116474    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:46.116626 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.116487    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:46.118498 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.118473    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-dshm" pod "a44120e6-3289-41db-b966-850b6f54431f" (UID: "a44120e6-3289-41db-b966-850b6f54431f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:46.118627 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.118561    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44120e6-3289-41db-b966-850b6f54431f-kube-api-access-m477z" pod "a44120e6-3289-41db-b966-850b6f54431f" (UID: "a44120e6-3289-41db-b966-850b6f54431f"). InnerVolumeSpecName "kube-api-access-m477z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:46.118627 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.118599    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44120e6-3289-41db-b966-850b6f54431f-tls-certs" pod "a44120e6-3289-41db-b966-850b6f54431f" (UID: "a44120e6-3289-41db-b966-850b6f54431f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:46.216804 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.216780    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44120e6-3289-41db-b966-850b6f54431f-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:46.216804 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.216804    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m477z\" (UniqueName: \"kubernetes.io/projected/a44120e6-3289-41db-b966-850b6f54431f-kube-api-access-m477z\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:46.216945 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.216814    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44120e6-3289-41db-b966-850b6f54431f-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:46.866041 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.866008    4853 generic.go:356] "Generic (PLEG): container finished" podID="a44120e6-3289-41db-b966-850b6f54431f" containerID="3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4" exitCode=0
Oct 30 18:53:46.866041 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.866043    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq" event={"ID":"a44120e6-3289-41db-b966-850b6f54431f","Type":"ContainerDied","Data":"3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4"}
Oct 30 18:53:46.866613 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.866065    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"
Oct 30 18:53:46.866613 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.866081    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq" event={"ID":"a44120e6-3289-41db-b966-850b6f54431f","Type":"ContainerDied","Data":"5c1cbb94086aa565d7269581ef6a45ffbc4da48378f224a54ebfd50f140e6e35"}
Oct 30 18:53:46.866613 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.866104    4853 scope.go:122] "RemoveContainer" containerID="3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4"
Oct 30 18:53:46.875356 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.875332    4853 scope.go:122] "RemoveContainer" containerID="3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4"
Oct 30 18:53:46.875634 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:53:46.875614    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4\": container with ID starting with 3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4 not found: ID does not exist" containerID="3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4"
Oct 30 18:53:46.875701 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.875643    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4"} err="failed to get container status \"3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4\": rpc error: code = NotFound desc = could not find container \"3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4\": container with ID starting with 3d3bd383b81d12c7e80014548096fdc84d63f3f7822ee9400d28a086dd50fbc4 not found: ID does not exist"
Oct 30 18:53:46.990877 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:46.990822    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:47.027362 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:47.027339    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"]
Oct 30 18:53:47.073917 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:47.073898    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76xb8dq"]
Oct 30 18:53:47.944255 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:47.944219    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44120e6-3289-41db-b966-850b6f54431f" path="/var/lib/kubelet/pods/a44120e6-3289-41db-b966-850b6f54431f/volumes"
Oct 30 18:53:49.872909 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:53:49.872864    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:53:53.710655 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:53.710620    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:58.657855 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:53:58.657799    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:19.872477 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:54:19.872441    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:54:33.996456 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:33.996422    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:34.213622 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:34.213560    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:38.175093 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.175058    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"]
Oct 30 18:54:38.175484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.175386    4853 cpu_manager.go:414] "containerMap: removing container" podUID="a44120e6-3289-41db-b966-850b6f54431f" containerName="main"
Oct 30 18:54:38.175484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.175397    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44120e6-3289-41db-b966-850b6f54431f" containerName="main"
Oct 30 18:54:38.175484 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.175457    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44120e6-3289-41db-b966-850b6f54431f" containerName="main"
Oct 30 18:54:38.178151 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.178107    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.231966 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.231945    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Oct 30 18:54:38.278608 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.278584    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Oct 30 18:54:38.278608 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.278584    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-4k49r\""
Oct 30 18:54:38.278775 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.278627    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Oct 30 18:54:38.280445 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.280419    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"]
Oct 30 18:54:38.300102 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.300078    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzk7l\" (UniqueName: \"kubernetes.io/projected/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-kube-api-access-nzk7l\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.300218 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.300116    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.300218 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.300134    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.300336 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.300277    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.300390 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.300331    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.300433 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.300413    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.377389 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.377358    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:38.402932 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.402887    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.403138 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.402964    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.403138 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.403055    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.403138 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.403087    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzk7l\" (UniqueName: \"kubernetes.io/projected/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-kube-api-access-nzk7l\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.403303 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.403138    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.403303 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.403171    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.407876 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.407834    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.408487 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.408462    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.409675 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.409642    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.414572 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.411586    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.414676 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.414614    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.463291 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.463233    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzk7l\" (UniqueName: \"kubernetes.io/projected/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-kube-api-access-nzk7l\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.502264 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.502244    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:38.871345 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.871306    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:38.872841 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:38.872820    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"]
Oct 30 18:54:38.879349 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:54:38.879325    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5490bc1b_51b6_41e1_bcb3_98c74fb3111b.slice/crio-6a7b17f107c38c92a482268b573eb846029156b918887bd106504b8515f2307f WatchSource:0}: Error finding container 6a7b17f107c38c92a482268b573eb846029156b918887bd106504b8515f2307f: Status 404 returned error can't find the container with id 6a7b17f107c38c92a482268b573eb846029156b918887bd106504b8515f2307f
Oct 30 18:54:39.120743 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:39.120697    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c" event={"ID":"5490bc1b-51b6-41e1-bcb3-98c74fb3111b","Type":"ContainerStarted","Data":"f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc"}
Oct 30 18:54:39.120947 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:39.120762    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c" event={"ID":"5490bc1b-51b6-41e1-bcb3-98c74fb3111b","Type":"ContainerStarted","Data":"6a7b17f107c38c92a482268b573eb846029156b918887bd106504b8515f2307f"}
Oct 30 18:54:39.236097 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:39.235993    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c" podStartSLOduration=1.235978834 podStartE2EDuration="1.235978834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:39.23387374 +0000 UTC m=+985.404230588" watchObservedRunningTime="2026-10-30 18:54:39.235978834 +0000 UTC m=+985.406335682"
Oct 30 18:54:39.502875 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:39.502805    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:41.687520 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:41.687482    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:45.658230 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:45.658190    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:48.171042 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:48.171008    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:48.388709 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:48.388678    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:48.503040 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:48.502960    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:48.515882 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:48.515859    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:49.185400 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:49.185367    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:54:49.873306 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:54:49.873264    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:54:54.003896 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:54:54.003858    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:18.393025 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.392992    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"]
Oct 30 18:55:18.402673 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:55:18.402635    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc" cmd=["/bin/sleep","15"]
Oct 30 18:55:18.402819 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.402690    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 18:55:18.402819 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:55:18.402703    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c" podUID="5490bc1b-51b6-41e1-bcb3-98c74fb3111b" containerName="main" containerID="cri-o://f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc"
Oct 30 18:55:18.402819 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.402749    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c" podUID="5490bc1b-51b6-41e1-bcb3-98c74fb3111b" containerName="main" containerID="cri-o://f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc" gracePeriod=60
Oct 30 18:55:18.496917 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.496884    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:18.708222 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.708189    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:18.754478 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.754458    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:55:18.806141 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.806114    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-dshm\") pod \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") "
Oct 30 18:55:18.806252 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.806184    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-home\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-home\") pod \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") "
Oct 30 18:55:18.806252 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.806227    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tls-certs\") pod \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") "
Oct 30 18:55:18.806336 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.806270    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-model-cache\") pod \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") "
Oct 30 18:55:18.806336 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.806329    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tmp-dir\") pod \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") "
Oct 30 18:55:18.806441 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.806365    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-kube-api-access-nzk7l\" (UniqueName: \"kubernetes.io/projected/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-kube-api-access-nzk7l\") pod \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\" (UID: \"5490bc1b-51b6-41e1-bcb3-98c74fb3111b\") "
Oct 30 18:55:18.806496 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.806458    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-home" pod "5490bc1b-51b6-41e1-bcb3-98c74fb3111b" (UID: "5490bc1b-51b6-41e1-bcb3-98c74fb3111b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:18.806641 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.806613    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-model-cache" pod "5490bc1b-51b6-41e1-bcb3-98c74fb3111b" (UID: "5490bc1b-51b6-41e1-bcb3-98c74fb3111b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:18.806641 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.806634    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:18.806641 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.806638    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tmp-dir" pod "5490bc1b-51b6-41e1-bcb3-98c74fb3111b" (UID: "5490bc1b-51b6-41e1-bcb3-98c74fb3111b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:18.808483 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.808449    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-dshm" pod "5490bc1b-51b6-41e1-bcb3-98c74fb3111b" (UID: "5490bc1b-51b6-41e1-bcb3-98c74fb3111b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:18.808483 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.808457    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tls-certs" pod "5490bc1b-51b6-41e1-bcb3-98c74fb3111b" (UID: "5490bc1b-51b6-41e1-bcb3-98c74fb3111b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:18.808616 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.808506    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-kube-api-access-nzk7l" pod "5490bc1b-51b6-41e1-bcb3-98c74fb3111b" (UID: "5490bc1b-51b6-41e1-bcb3-98c74fb3111b"). InnerVolumeSpecName "kube-api-access-nzk7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:18.907346 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.907325    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:18.907346 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.907347    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:18.907494 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.907357    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:18.907494 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.907366    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzk7l\" (UniqueName: \"kubernetes.io/projected/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-kube-api-access-nzk7l\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:18.907494 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:18.907376    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5490bc1b-51b6-41e1-bcb3-98c74fb3111b-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:19.318977 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:19.318944    4853 generic.go:356] "Generic (PLEG): container finished" podID="5490bc1b-51b6-41e1-bcb3-98c74fb3111b" containerID="f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc" exitCode=0
Oct 30 18:55:19.319128 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:19.318999    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"
Oct 30 18:55:19.319128 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:19.319031    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c" event={"ID":"5490bc1b-51b6-41e1-bcb3-98c74fb3111b","Type":"ContainerDied","Data":"f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc"}
Oct 30 18:55:19.319128 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:19.319082    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c" event={"ID":"5490bc1b-51b6-41e1-bcb3-98c74fb3111b","Type":"ContainerDied","Data":"6a7b17f107c38c92a482268b573eb846029156b918887bd106504b8515f2307f"}
Oct 30 18:55:19.319128 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:19.319105    4853 scope.go:122] "RemoveContainer" containerID="f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc"
Oct 30 18:55:19.335627 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:19.335606    4853 scope.go:122] "RemoveContainer" containerID="f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc"
Oct 30 18:55:19.335906 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:55:19.335884    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc\": container with ID starting with f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc not found: ID does not exist" containerID="f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc"
Oct 30 18:55:19.335961 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:19.335916    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc"} err="failed to get container status \"f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc\": rpc error: code = NotFound desc = could not find container \"f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc\": container with ID starting with f802a53bc1cea623e0820046c08a16aea01e4d4510728066bd1c89c1342df3fc not found: ID does not exist"
Oct 30 18:55:19.478474 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:19.478449    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"]
Oct 30 18:55:19.483994 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:19.483972    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-hjx6c"]
Oct 30 18:55:19.873710 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:55:19.873676    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:55:19.939343 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:19.939313    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5490bc1b-51b6-41e1-bcb3-98c74fb3111b" path="/var/lib/kubelet/pods/5490bc1b-51b6-41e1-bcb3-98c74fb3111b/volumes"
Oct 30 18:55:24.490486 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:24.490456    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:49.875267 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:55:49.875224    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:55:54.328890 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:55:54.328840    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:06.635322 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:06.635288    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:06.879487 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:06.879430    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:14.202031 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:14.201991    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:15.455697 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.455665    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"]
Oct 30 18:56:15.456070 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.455976    4853 cpu_manager.go:414] "containerMap: removing container" podUID="5490bc1b-51b6-41e1-bcb3-98c74fb3111b" containerName="main"
Oct 30 18:56:15.456070 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.455986    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5490bc1b-51b6-41e1-bcb3-98c74fb3111b" containerName="main"
Oct 30 18:56:15.456070 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.456039    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="5490bc1b-51b6-41e1-bcb3-98c74fb3111b" containerName="main"
Oct 30 18:56:15.460121 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.460102    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.549317 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.549292    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Oct 30 18:56:15.549442 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.549358    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-x4x8q\""
Oct 30 18:56:15.557207 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.557190    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Oct 30 18:56:15.557342 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.557325    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Oct 30 18:56:15.559349 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.559333    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"]
Oct 30 18:56:15.590856 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.590835    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:15.627641 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.627620    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.627759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.627650    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-dshm\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.627759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.627671    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-home\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.627759 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.627694    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.627886 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.627814    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzr2s\" (UniqueName: \"kubernetes.io/projected/94d09f64-5882-4e80-91a3-01952a4e65a6-kube-api-access-jzr2s\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.627886 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.627846    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d09f64-5882-4e80-91a3-01952a4e65a6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.728926 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.728851    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.728926 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.728890    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-dshm\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.729114 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.729023    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-home\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.729114 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.729068    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.729193 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.729128    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzr2s\" (UniqueName: \"kubernetes.io/projected/94d09f64-5882-4e80-91a3-01952a4e65a6-kube-api-access-jzr2s\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.729193 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.729174    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d09f64-5882-4e80-91a3-01952a4e65a6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.729354 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.729334    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.729413 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.729374    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-home\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.729457 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.729441    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.731355 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.731328    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-dshm\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.731639 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.731618    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d09f64-5882-4e80-91a3-01952a4e65a6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:15.783872 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:15.783853    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzr2s\" (UniqueName: \"kubernetes.io/projected/94d09f64-5882-4e80-91a3-01952a4e65a6-kube-api-access-jzr2s\") pod \"scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:16.074257 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:16.074180    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:16.131152 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:16.131125    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:16.473050 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:16.473021    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"]
Oct 30 18:56:16.478799 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:56:16.478768    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d09f64_5882_4e80_91a3_01952a4e65a6.slice/crio-d599d23f8f7e71416e511d659cba9421b831a354adddb78c755d4974f4592976 WatchSource:0}: Error finding container d599d23f8f7e71416e511d659cba9421b831a354adddb78c755d4974f4592976: Status 404 returned error can't find the container with id d599d23f8f7e71416e511d659cba9421b831a354adddb78c755d4974f4592976
Oct 30 18:56:16.589505 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:16.589469    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl" event={"ID":"94d09f64-5882-4e80-91a3-01952a4e65a6","Type":"ContainerStarted","Data":"3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14"}
Oct 30 18:56:16.589658 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:16.589572    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl" event={"ID":"94d09f64-5882-4e80-91a3-01952a4e65a6","Type":"ContainerStarted","Data":"d599d23f8f7e71416e511d659cba9421b831a354adddb78c755d4974f4592976"}
Oct 30 18:56:16.700485 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:16.700441    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl" podStartSLOduration=1.700428371 podStartE2EDuration="1.700428371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:16.698822344 +0000 UTC m=+1082.869179205" watchObservedRunningTime="2026-10-30 18:56:16.700428371 +0000 UTC m=+1082.870785218"
Oct 30 18:56:17.074303 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:17.074264    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:19.872954 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:56:19.872910    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:56:21.364588 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:21.364558    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:21.621752 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:21.621678    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:26.074596 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:26.074563    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:26.086987 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:26.086961    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:26.643936 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:26.643909    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:30.464175 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:30.464134    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:48.174972 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:48.174924    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:48.387365 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:48.387334    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:49.872453 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:56:49.872418    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:56:54.605254 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:54.605213    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"]
Oct 30 18:56:54.615098 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:56:54.615067    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14" cmd=["/bin/sleep","15"]
Oct 30 18:56:54.615221 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:54.615116    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 18:56:54.615221 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:56:54.615128    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl" podUID="94d09f64-5882-4e80-91a3-01952a4e65a6" containerName="main" containerID="cri-o://3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14"
Oct 30 18:56:54.615221 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:54.615162    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl" podUID="94d09f64-5882-4e80-91a3-01952a4e65a6" containerName="main" containerID="cri-o://3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14" gracePeriod=60
Oct 30 18:56:54.633727 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:54.633700    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl" podUID="94d09f64-5882-4e80-91a3-01952a4e65a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Oct 30 18:56:54.656352 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:54.656255    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:54.909801 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:54.909710    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:55.010095 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.010072    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:55.030190 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.030169    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-home\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-home\") pod \"94d09f64-5882-4e80-91a3-01952a4e65a6\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") "
Oct 30 18:56:55.030312 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.030229    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-dshm\") pod \"94d09f64-5882-4e80-91a3-01952a4e65a6\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") "
Oct 30 18:56:55.030312 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.030260    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d09f64-5882-4e80-91a3-01952a4e65a6-kube-api-access-jzr2s\" (UniqueName: \"kubernetes.io/projected/94d09f64-5882-4e80-91a3-01952a4e65a6-kube-api-access-jzr2s\") pod \"94d09f64-5882-4e80-91a3-01952a4e65a6\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") "
Oct 30 18:56:55.030312 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.030297    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-model-cache\") pod \"94d09f64-5882-4e80-91a3-01952a4e65a6\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") "
Oct 30 18:56:55.030483 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.030395    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d09f64-5882-4e80-91a3-01952a4e65a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/94d09f64-5882-4e80-91a3-01952a4e65a6-tls-certs\") pod \"94d09f64-5882-4e80-91a3-01952a4e65a6\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") "
Oct 30 18:56:55.030483 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.030424    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-home" pod "94d09f64-5882-4e80-91a3-01952a4e65a6" (UID: "94d09f64-5882-4e80-91a3-01952a4e65a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:55.030483 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.030451    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-tmp-dir\") pod \"94d09f64-5882-4e80-91a3-01952a4e65a6\" (UID: \"94d09f64-5882-4e80-91a3-01952a4e65a6\") "
Oct 30 18:56:55.030661 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.030487    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-model-cache" pod "94d09f64-5882-4e80-91a3-01952a4e65a6" (UID: "94d09f64-5882-4e80-91a3-01952a4e65a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:55.030716 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.030696    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-tmp-dir" pod "94d09f64-5882-4e80-91a3-01952a4e65a6" (UID: "94d09f64-5882-4e80-91a3-01952a4e65a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:55.030783 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.030762    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:55.030893 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.030786    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:55.032657 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.032627    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-dshm" pod "94d09f64-5882-4e80-91a3-01952a4e65a6" (UID: "94d09f64-5882-4e80-91a3-01952a4e65a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:56:55.032740 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.032666    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d09f64-5882-4e80-91a3-01952a4e65a6-kube-api-access-jzr2s" pod "94d09f64-5882-4e80-91a3-01952a4e65a6" (UID: "94d09f64-5882-4e80-91a3-01952a4e65a6"). InnerVolumeSpecName "kube-api-access-jzr2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:55.032783 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.032754    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d09f64-5882-4e80-91a3-01952a4e65a6-tls-certs" pod "94d09f64-5882-4e80-91a3-01952a4e65a6" (UID: "94d09f64-5882-4e80-91a3-01952a4e65a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:55.131878 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.131853    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzr2s\" (UniqueName: \"kubernetes.io/projected/94d09f64-5882-4e80-91a3-01952a4e65a6-kube-api-access-jzr2s\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:55.131878 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.131879    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d09f64-5882-4e80-91a3-01952a4e65a6-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:55.132008 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.131889    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:55.132008 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.131896    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d09f64-5882-4e80-91a3-01952a4e65a6-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:55.771832 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.771799    4853 generic.go:356] "Generic (PLEG): container finished" podID="94d09f64-5882-4e80-91a3-01952a4e65a6" containerID="3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14" exitCode=0
Oct 30 18:56:55.772343 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.771842    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"
Oct 30 18:56:55.772343 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.771881    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl" event={"ID":"94d09f64-5882-4e80-91a3-01952a4e65a6","Type":"ContainerDied","Data":"3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14"}
Oct 30 18:56:55.772343 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.771929    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl" event={"ID":"94d09f64-5882-4e80-91a3-01952a4e65a6","Type":"ContainerDied","Data":"d599d23f8f7e71416e511d659cba9421b831a354adddb78c755d4974f4592976"}
Oct 30 18:56:55.772343 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.771952    4853 scope.go:122] "RemoveContainer" containerID="3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14"
Oct 30 18:56:55.781459 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.781439    4853 scope.go:122] "RemoveContainer" containerID="3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14"
Oct 30 18:56:55.781778 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:56:55.781761    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14\": container with ID starting with 3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14 not found: ID does not exist" containerID="3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14"
Oct 30 18:56:55.781843 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.781787    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14"} err="failed to get container status \"3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14\": rpc error: code = NotFound desc = could not find container \"3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14\": container with ID starting with 3ba1d5ada7fed6cea670b311b17d171ce71a0eda36507c555db9679917fdda14 not found: ID does not exist"
Oct 30 18:56:55.929685 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.929660    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"]
Oct 30 18:56:55.976271 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:55.976246    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-74cd6f7448-l4ksl"]
Oct 30 18:56:57.939123 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:56:57.939076    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d09f64-5882-4e80-91a3-01952a4e65a6" path="/var/lib/kubelet/pods/94d09f64-5882-4e80-91a3-01952a4e65a6/volumes"
Oct 30 18:57:19.875921 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:57:19.875881    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:57:37.633663 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:37.633623    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:37.888762 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:37.888680    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:42.934719 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:42.934688    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:43.209377 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:43.209278    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:45.891580 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:45.891518    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:46.452262 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:46.452214    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:46.768166 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:46.768088    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:49.334226 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.334188    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"]
Oct 30 18:57:49.334714 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.334694    4853 cpu_manager.go:414] "containerMap: removing container" podUID="94d09f64-5882-4e80-91a3-01952a4e65a6" containerName="main"
Oct 30 18:57:49.334760 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.334718    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d09f64-5882-4e80-91a3-01952a4e65a6" containerName="main"
Oct 30 18:57:49.334838 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.334826    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d09f64-5882-4e80-91a3-01952a4e65a6" containerName="main"
Oct 30 18:57:49.337881 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.337862    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.397385 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.397360    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"]
Oct 30 18:57:49.430028 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.429985    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.430164 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.430028    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.430164 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.430060    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.430275 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.430208    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.430275 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.430239    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvwf\" (UniqueName: \"kubernetes.io/projected/0faad873-794a-4f22-a1b7-4480dfe6377d-kube-api-access-9jvwf\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.430378 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.430307    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0faad873-794a-4f22-a1b7-4480dfe6377d-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.435723 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.435702    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Oct 30 18:57:49.435723 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.435711    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-fbc2j\""
Oct 30 18:57:49.435867 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.435754    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Oct 30 18:57:49.435867 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.435775    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Oct 30 18:57:49.479783 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.479762    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:49.531205 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.531184    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0faad873-794a-4f22-a1b7-4480dfe6377d-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.531304 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.531232    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.531304 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.531251    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.531304 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.531280    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.531454 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.531319    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.531454 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.531433    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvwf\" (UniqueName: \"kubernetes.io/projected/0faad873-794a-4f22-a1b7-4480dfe6377d-kube-api-access-9jvwf\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.531678 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.531658    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.531757 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.531712    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.531813 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.531762    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.533685 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.533663    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.534059 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.534042    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0faad873-794a-4f22-a1b7-4480dfe6377d-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.588050 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.587981    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvwf\" (UniqueName: \"kubernetes.io/projected/0faad873-794a-4f22-a1b7-4480dfe6377d-kube-api-access-9jvwf\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-ghgmk\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:49.652033 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:49.651726    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:50.095741 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:57:50.087792    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:57:50.115554 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:50.111994    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:50.572881 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:50.570775    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"]
Oct 30 18:57:51.151619 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:51.151566    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk" event={"ID":"0faad873-794a-4f22-a1b7-4480dfe6377d","Type":"ContainerStarted","Data":"c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6"}
Oct 30 18:57:51.151902 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:51.151881    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk" event={"ID":"0faad873-794a-4f22-a1b7-4480dfe6377d","Type":"ContainerStarted","Data":"f2d317ae78e1b368ec18e36d6fce1f1e958689fa2bad6a1d86f0ff5ee68df0bf"}
Oct 30 18:57:51.265229 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:51.265165    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk" podStartSLOduration=2.265142839 podStartE2EDuration="2.265142839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:51.264916214 +0000 UTC m=+1177.435273062" watchObservedRunningTime="2026-10-30 18:57:51.265142839 +0000 UTC m=+1177.435499691"
Oct 30 18:57:51.541707 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:51.541584    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:51.652678 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:51.652595    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:53.624557 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:53.620399    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:53.850024 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:53.849948    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:55.111034 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:55.110940    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:57.586668 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:57.586631    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:58.315961 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:58.315907    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:58.559160 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:58.559129    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:59.652903 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:59.652870    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:57:59.665824 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:57:59.665803    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:58:00.309341 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:00.309316    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:58:06.676132 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:06.676092    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:06.899335 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:06.899303    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:19.872891 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:58:19.872853    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:58:29.601697 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:29.601657    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:29.861762 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:29.861683    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:30.537508 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.537477    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"]
Oct 30 18:58:30.547081 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:58:30.547044    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6" cmd=["/bin/sleep","15"]
Oct 30 18:58:30.547227 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.547097    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 18:58:30.547227 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:58:30.547109    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk" podUID="0faad873-794a-4f22-a1b7-4480dfe6377d" containerName="main" containerID="cri-o://c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6"
Oct 30 18:58:30.547227 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.547157    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk" podUID="0faad873-794a-4f22-a1b7-4480dfe6377d" containerName="main" containerID="cri-o://c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6" gracePeriod=60
Oct 30 18:58:30.610612 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.610584    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:30.813033 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.813009    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:58:30.849089 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.849063    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:30.870664 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.870644    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-home\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-home\") pod \"0faad873-794a-4f22-a1b7-4480dfe6377d\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") "
Oct 30 18:58:30.870755 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.870680    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-tmp-dir\") pod \"0faad873-794a-4f22-a1b7-4480dfe6377d\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") "
Oct 30 18:58:30.870755 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.870742    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-dshm\") pod \"0faad873-794a-4f22-a1b7-4480dfe6377d\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") "
Oct 30 18:58:30.870906 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.870759    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0faad873-794a-4f22-a1b7-4480dfe6377d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0faad873-794a-4f22-a1b7-4480dfe6377d-tls-certs\") pod \"0faad873-794a-4f22-a1b7-4480dfe6377d\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") "
Oct 30 18:58:30.870906 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.870800    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0faad873-794a-4f22-a1b7-4480dfe6377d-kube-api-access-9jvwf\" (UniqueName: \"kubernetes.io/projected/0faad873-794a-4f22-a1b7-4480dfe6377d-kube-api-access-9jvwf\") pod \"0faad873-794a-4f22-a1b7-4480dfe6377d\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") "
Oct 30 18:58:30.870906 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.870831    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-model-cache\") pod \"0faad873-794a-4f22-a1b7-4480dfe6377d\" (UID: \"0faad873-794a-4f22-a1b7-4480dfe6377d\") "
Oct 30 18:58:30.871081 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.870930    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-home" pod "0faad873-794a-4f22-a1b7-4480dfe6377d" (UID: "0faad873-794a-4f22-a1b7-4480dfe6377d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:30.871081 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.871016    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-tmp-dir" pod "0faad873-794a-4f22-a1b7-4480dfe6377d" (UID: "0faad873-794a-4f22-a1b7-4480dfe6377d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:30.871081 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.871068    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:30.871261 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.871212    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-model-cache" pod "0faad873-794a-4f22-a1b7-4480dfe6377d" (UID: "0faad873-794a-4f22-a1b7-4480dfe6377d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:30.873015 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.872995    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-dshm" pod "0faad873-794a-4f22-a1b7-4480dfe6377d" (UID: "0faad873-794a-4f22-a1b7-4480dfe6377d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:30.873131 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.873088    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0faad873-794a-4f22-a1b7-4480dfe6377d-tls-certs" pod "0faad873-794a-4f22-a1b7-4480dfe6377d" (UID: "0faad873-794a-4f22-a1b7-4480dfe6377d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:30.873178 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.873158    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0faad873-794a-4f22-a1b7-4480dfe6377d-kube-api-access-9jvwf" pod "0faad873-794a-4f22-a1b7-4480dfe6377d" (UID: "0faad873-794a-4f22-a1b7-4480dfe6377d"). InnerVolumeSpecName "kube-api-access-9jvwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:30.971576 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.971552    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvwf\" (UniqueName: \"kubernetes.io/projected/0faad873-794a-4f22-a1b7-4480dfe6377d-kube-api-access-9jvwf\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:30.971576 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.971578    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:30.971717 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.971608    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:30.971717 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.971618    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0faad873-794a-4f22-a1b7-4480dfe6377d-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:30.971717 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:30.971631    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0faad873-794a-4f22-a1b7-4480dfe6377d-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:31.457778 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:31.457749    4853 generic.go:356] "Generic (PLEG): container finished" podID="0faad873-794a-4f22-a1b7-4480dfe6377d" containerID="c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6" exitCode=0
Oct 30 18:58:31.457909 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:31.457801    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"
Oct 30 18:58:31.457909 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:31.457840    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk" event={"ID":"0faad873-794a-4f22-a1b7-4480dfe6377d","Type":"ContainerDied","Data":"c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6"}
Oct 30 18:58:31.457909 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:31.457893    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk" event={"ID":"0faad873-794a-4f22-a1b7-4480dfe6377d","Type":"ContainerDied","Data":"f2d317ae78e1b368ec18e36d6fce1f1e958689fa2bad6a1d86f0ff5ee68df0bf"}
Oct 30 18:58:31.458014 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:31.457924    4853 scope.go:122] "RemoveContainer" containerID="c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6"
Oct 30 18:58:31.469284 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:31.469265    4853 scope.go:122] "RemoveContainer" containerID="c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6"
Oct 30 18:58:31.469743 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:58:31.469718    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6\": container with ID starting with c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6 not found: ID does not exist" containerID="c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6"
Oct 30 18:58:31.469840 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:31.469751    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6"} err="failed to get container status \"c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6\": rpc error: code = NotFound desc = could not find container \"c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6\": container with ID starting with c38b82ab3f29adcc536ef13be5e6c263c3a72b6619bcd95ac97af88f164f6cb6 not found: ID does not exist"
Oct 30 18:58:31.616719 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:31.616693    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"]
Oct 30 18:58:31.662560 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:31.662517    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-ghgmk"]
Oct 30 18:58:31.939017 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:31.938979    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0faad873-794a-4f22-a1b7-4480dfe6377d" path="/var/lib/kubelet/pods/0faad873-794a-4f22-a1b7-4480dfe6377d/volumes"
Oct 30 18:58:49.875970 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:58:49.875932    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:58:55.543414 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:58:55.543375    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:18.338478 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:18.338386    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:18.576512 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:18.576482    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:18.894431 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:18.894404    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:19.113086 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:19.113060    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:19.872898 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:59:19.872865    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:59:26.627826 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.627788    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:26.940761 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.940692    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"]
Oct 30 18:59:26.941024 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.941012    4853 cpu_manager.go:414] "containerMap: removing container" podUID="0faad873-794a-4f22-a1b7-4480dfe6377d" containerName="main"
Oct 30 18:59:26.941108 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.941026    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0faad873-794a-4f22-a1b7-4480dfe6377d" containerName="main"
Oct 30 18:59:26.941108 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.941079    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="0faad873-794a-4f22-a1b7-4480dfe6377d" containerName="main"
Oct 30 18:59:26.943765 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.943745    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:26.986607 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.986578    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:26.986738 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.986622    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:26.986738 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.986695    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:26.986816 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.986756    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:26.986816 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.986780    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:26.986816 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:26.986801    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddx9d\" (UniqueName: \"kubernetes.io/projected/ed204677-a0b8-48fd-8f6b-71a9466aa46b-kube-api-access-ddx9d\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.042956 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.042913    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Oct 30 18:59:27.043111 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.043092    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Oct 30 18:59:27.043199 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.043177    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"]
Oct 30 18:59:27.044686 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.044671    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-bvgsz\""
Oct 30 18:59:27.044875 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.044859    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Oct 30 18:59:27.087125 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.087101    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.087255 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.087145    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.087255 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.087172    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.087255 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.087206    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.087255 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.087228    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddx9d\" (UniqueName: \"kubernetes.io/projected/ed204677-a0b8-48fd-8f6b-71a9466aa46b-kube-api-access-ddx9d\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.087455 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.087254    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.087598 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.087578    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.087677 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.087637    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.087677 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.087662    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.089691 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.089671    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.089965 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.089946    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.124117 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.124095    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:27.142596 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.142571    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddx9d\" (UniqueName: \"kubernetes.io/projected/ed204677-a0b8-48fd-8f6b-71a9466aa46b-kube-api-access-ddx9d\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.257242 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.257186    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:27.496106 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.496072    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:27.640648 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.640618    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"]
Oct 30 18:59:27.648044 ip-10-0-129-31 kubenswrapper[4853]: W1030 18:59:27.648012    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded204677_a0b8_48fd_8f6b_71a9466aa46b.slice/crio-a4d570116650b6a309452d08345bd5ec445e2afc7463537b81fc1efd532fd0e4 WatchSource:0}: Error finding container a4d570116650b6a309452d08345bd5ec445e2afc7463537b81fc1efd532fd0e4: Status 404 returned error can't find the container with id a4d570116650b6a309452d08345bd5ec445e2afc7463537b81fc1efd532fd0e4
Oct 30 18:59:27.701183 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.701153    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:27.748967 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.748918    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q" event={"ID":"ed204677-a0b8-48fd-8f6b-71a9466aa46b","Type":"ContainerStarted","Data":"12f07270eedaba19411723c5f858b5717695dff5adc5f2a6f5eb76781669a584"}
Oct 30 18:59:27.749147 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.748983    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q" event={"ID":"ed204677-a0b8-48fd-8f6b-71a9466aa46b","Type":"ContainerStarted","Data":"a4d570116650b6a309452d08345bd5ec445e2afc7463537b81fc1efd532fd0e4"}
Oct 30 18:59:27.860398 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:27.860347    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q" podStartSLOduration=1.860333898 podStartE2EDuration="1.860333898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:27.858272107 +0000 UTC m=+1274.028628955" watchObservedRunningTime="2026-10-30 18:59:27.860333898 +0000 UTC m=+1274.030690745"
Oct 30 18:59:28.257743 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:28.257711    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:28.348742 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:28.348661    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:28.570702 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:28.570621    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:28.776465 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:28.776432    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:34.471553 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:34.470978    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:34.741634 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:34.737362    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:35.153259 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:35.153224    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:35.353382 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:35.353347    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:37.205620 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:37.205458    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:37.267696 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:37.267594    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:37.318148 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:37.317843    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:37.900838 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:37.900807    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 18:59:40.725155 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:40.724849    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:49.874162 ip-10-0-129-31 kubenswrapper[4853]: E1030 18:59:49.874127    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 18:59:55.872204 ip-10-0-129-31 kubenswrapper[4853]: I1030 18:59:55.872144    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:07.559587 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:07.559554    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"]
Oct 30 19:00:07.569878 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:00:07.569838    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="12f07270eedaba19411723c5f858b5717695dff5adc5f2a6f5eb76781669a584" cmd=["/bin/sleep","15"]
Oct 30 19:00:07.570050 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:07.569895    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:00:07.570050 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:00:07.569908    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q" podUID="ed204677-a0b8-48fd-8f6b-71a9466aa46b" containerName="main" containerID="cri-o://12f07270eedaba19411723c5f858b5717695dff5adc5f2a6f5eb76781669a584"
Oct 30 19:00:07.570050 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:07.569953    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q" podUID="ed204677-a0b8-48fd-8f6b-71a9466aa46b" containerName="main" containerID="cri-o://12f07270eedaba19411723c5f858b5717695dff5adc5f2a6f5eb76781669a584" gracePeriod=60
Oct 30 19:00:07.642813 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:07.642779    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:07.867071 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:07.867030    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q" podUID="ed204677-a0b8-48fd-8f6b-71a9466aa46b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Oct 30 19:00:07.869026 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:07.869004    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:08.003460 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.003428    4853 generic.go:356] "Generic (PLEG): container finished" podID="ed204677-a0b8-48fd-8f6b-71a9466aa46b" containerID="12f07270eedaba19411723c5f858b5717695dff5adc5f2a6f5eb76781669a584" exitCode=0
Oct 30 19:00:08.003460 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.003461    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q" event={"ID":"ed204677-a0b8-48fd-8f6b-71a9466aa46b","Type":"ContainerDied","Data":"12f07270eedaba19411723c5f858b5717695dff5adc5f2a6f5eb76781669a584"}
Oct 30 19:00:08.309567 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.309540    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 19:00:08.379989 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.379952    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-dshm\") pod \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") "
Oct 30 19:00:08.379989 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.379994    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-model-cache\") pod \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") "
Oct 30 19:00:08.380123 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.380036    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tls-certs\") pod \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") "
Oct 30 19:00:08.380123 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.380104    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed204677-a0b8-48fd-8f6b-71a9466aa46b-kube-api-access-ddx9d\" (UniqueName: \"kubernetes.io/projected/ed204677-a0b8-48fd-8f6b-71a9466aa46b-kube-api-access-ddx9d\") pod \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") "
Oct 30 19:00:08.380189 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.380167    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tmp-dir\") pod \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") "
Oct 30 19:00:08.380239 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.380194    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-home\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-home\") pod \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\" (UID: \"ed204677-a0b8-48fd-8f6b-71a9466aa46b\") "
Oct 30 19:00:08.380331 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.380312    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-model-cache" pod "ed204677-a0b8-48fd-8f6b-71a9466aa46b" (UID: "ed204677-a0b8-48fd-8f6b-71a9466aa46b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:08.380471 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.380452    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:08.380471 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.380436    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tmp-dir" pod "ed204677-a0b8-48fd-8f6b-71a9466aa46b" (UID: "ed204677-a0b8-48fd-8f6b-71a9466aa46b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:08.380606 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.380499    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-home" pod "ed204677-a0b8-48fd-8f6b-71a9466aa46b" (UID: "ed204677-a0b8-48fd-8f6b-71a9466aa46b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:08.382554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.382515    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed204677-a0b8-48fd-8f6b-71a9466aa46b-kube-api-access-ddx9d" pod "ed204677-a0b8-48fd-8f6b-71a9466aa46b" (UID: "ed204677-a0b8-48fd-8f6b-71a9466aa46b"). InnerVolumeSpecName "kube-api-access-ddx9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:08.382717 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.382695    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-dshm" pod "ed204677-a0b8-48fd-8f6b-71a9466aa46b" (UID: "ed204677-a0b8-48fd-8f6b-71a9466aa46b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:08.382759 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.382700    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tls-certs" pod "ed204677-a0b8-48fd-8f6b-71a9466aa46b" (UID: "ed204677-a0b8-48fd-8f6b-71a9466aa46b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:08.481867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.481812    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:08.481867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.481834    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:08.481867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.481845    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddx9d\" (UniqueName: \"kubernetes.io/projected/ed204677-a0b8-48fd-8f6b-71a9466aa46b-kube-api-access-ddx9d\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:08.481867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.481855    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:08.481867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:08.481864    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed204677-a0b8-48fd-8f6b-71a9466aa46b-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:09.009602 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:09.009574    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"
Oct 30 19:00:09.009602 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:09.009589    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q" event={"ID":"ed204677-a0b8-48fd-8f6b-71a9466aa46b","Type":"ContainerDied","Data":"a4d570116650b6a309452d08345bd5ec445e2afc7463537b81fc1efd532fd0e4"}
Oct 30 19:00:09.010106 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:09.009657    4853 scope.go:122] "RemoveContainer" containerID="12f07270eedaba19411723c5f858b5717695dff5adc5f2a6f5eb76781669a584"
Oct 30 19:00:09.166924 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:09.166897    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"]
Oct 30 19:00:09.213313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:09.213286    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-b8f4q"]
Oct 30 19:00:09.938476 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:09.938435    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed204677-a0b8-48fd-8f6b-71a9466aa46b" path="/var/lib/kubelet/pods/ed204677-a0b8-48fd-8f6b-71a9466aa46b/volumes"
Oct 30 19:00:14.098896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:14.098835    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:14.320006 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:14.319968    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:19.875305 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:00:19.875271    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:00:49.874878 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:00:49.874835    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:00:55.075936 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:55.075906    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:55.328756 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:55.328673    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:56.180966 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:00:56.180933    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:00.735587 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:00.735548    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:00.946168 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:00.946142    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:01.470771 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:01.470726    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:01.754080 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:01.754011    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:03.415922 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:03.415888    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:05.851195 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:05.851149    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:07.644436 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.644404    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"]
Oct 30 19:01:07.644849 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.644747    4853 cpu_manager.go:414] "containerMap: removing container" podUID="ed204677-a0b8-48fd-8f6b-71a9466aa46b" containerName="main"
Oct 30 19:01:07.644849 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.644760    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed204677-a0b8-48fd-8f6b-71a9466aa46b" containerName="main"
Oct 30 19:01:07.644849 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.644814    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed204677-a0b8-48fd-8f6b-71a9466aa46b" containerName="main"
Oct 30 19:01:07.647614 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.647594    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.702033 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.702011    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Oct 30 19:01:07.707294 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.707266    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k589n\" (UniqueName: \"kubernetes.io/projected/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-kube-api-access-k589n\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.707424 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.707308    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.707424 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.707326    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.707424 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.707387    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.707637 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.707439    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.707637 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.707489    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.747782 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.747759    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Oct 30 19:01:07.747887 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.747847    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-9w5px\""
Oct 30 19:01:07.747887 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.747872    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-5zbs6\""
Oct 30 19:01:07.747995 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.747871    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Oct 30 19:01:07.751046 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.751029    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"]
Oct 30 19:01:07.808313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.808291    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k589n\" (UniqueName: \"kubernetes.io/projected/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-kube-api-access-k589n\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.808425 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.808331    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.808425 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.808350    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.808573 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.808551    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.808653 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.808593    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.808710 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.808696    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.808765 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.808724    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.808880 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.808857    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.808938 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.808906    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.811643 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.811622    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.811899 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.811877    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.865896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.865873    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k589n\" (UniqueName: \"kubernetes.io/projected/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-kube-api-access-k589n\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:07.921674 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.921600    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:07.961730 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:07.961694    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:08.392549 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:08.392500    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"]
Oct 30 19:01:08.399197 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:01:08.399172    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce7050b_1bdb_4d5c_b74c_33f7ac2a00c0.slice/crio-09cb75fe3b3bb1e37330d913bf6ac5c79f3cb52be7a20957b62c0e276fc63722 WatchSource:0}: Error finding container 09cb75fe3b3bb1e37330d913bf6ac5c79f3cb52be7a20957b62c0e276fc63722: Status 404 returned error can't find the container with id 09cb75fe3b3bb1e37330d913bf6ac5c79f3cb52be7a20957b62c0e276fc63722
Oct 30 19:01:08.401495 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:08.401476    4853 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:01:08.615561 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:08.615508    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:09.293504 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:09.293471    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" event={"ID":"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0","Type":"ContainerStarted","Data":"09cb75fe3b3bb1e37330d913bf6ac5c79f3cb52be7a20957b62c0e276fc63722"}
Oct 30 19:01:09.846523 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:09.846488    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:10.903198 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:10.903158    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:11.134881 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:11.134848    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:11.305283 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:11.305214    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" event={"ID":"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0","Type":"ContainerStarted","Data":"e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161"}
Oct 30 19:01:11.305418 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:11.305298    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:11.701034 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:11.701003    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:12.338554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:12.338230    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" event={"ID":"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0","Type":"ContainerStarted","Data":"f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f"}
Oct 30 19:01:12.464554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:12.463615    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" podStartSLOduration=3.537125254 podStartE2EDuration="5.463594206s" totalImagesPullingTime="1.926468952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:12.456959934 +0000 UTC m=+1378.627316784" watchObservedRunningTime="2026-10-30 19:01:12.463594206 +0000 UTC m=+1378.633951053"
Oct 30 19:01:12.965698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:12.965657    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:15.396554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:15.393283    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:15.666596 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:15.666490    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:15.988630 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:15.988523    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"]
Oct 30 19:01:16.002554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:15.998810    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.044603 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.044568    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"]
Oct 30 19:01:16.049946 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.049923    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Oct 30 19:01:16.052557 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.050189    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:01:16.052813 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.052798    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Oct 30 19:01:16.052975 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.052962    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-tzt9r\""
Oct 30 19:01:16.094315 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.094283    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-envoy\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.098589 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.094520    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9df7298-84bf-4bec-b65b-6b034f106ed4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.098814 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.098793    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-token\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.098962 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.098943    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-credential-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.099133 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.099116    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-podinfo\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.099254 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.099239    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqkzf\" (UniqueName: \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-kube-api-access-gqkzf\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.099378 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.099363    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-certs\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.099514 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.099499    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-data\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.103721 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.103699    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.235291 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.235253    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-token\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.236439 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.236419    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-credential-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.237049 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.237032    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-podinfo\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.240746 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.240689    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqkzf\" (UniqueName: \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-kube-api-access-gqkzf\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.244918 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.244897    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-certs\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.248953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.248933    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-data\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.266644 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.266621    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.270671 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.270647    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-envoy\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.276987 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.276968    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9df7298-84bf-4bec-b65b-6b034f106ed4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.305877 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.250874    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-data\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.309551 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.267083    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.309718 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.236344    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:16.313549 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.248880    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-certs\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.313703 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.236956    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-credential-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.330629 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.294220    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-podinfo\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.330629 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.300027    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9df7298-84bf-4bec-b65b-6b034f106ed4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.336286 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.336236    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-envoy\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.348564 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.348437    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqkzf\" (UniqueName: \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-kube-api-access-gqkzf\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.435555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.422185    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-token\") pod \"router-gateway-1-openshift-default-745b597ff7-zg4kq\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:16.537361 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.537238    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:16.635160 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:16.635130    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:17.377352 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:17.376948    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"]
Oct 30 19:01:17.448815 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:17.448029    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:01:17.448815 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:17.448121    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:01:17.448815 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:17.448165    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:01:17.563121 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:17.563031    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq" event={"ID":"e9df7298-84bf-4bec-b65b-6b034f106ed4","Type":"ContainerStarted","Data":"3df786d8e97cc225b6b1a4a8711b80c4a00619e813e522c60216a867183e5783"}
Oct 30 19:01:17.971608 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:17.970442    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:17.997150 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:17.996863    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:18.588744 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:18.588548    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq" event={"ID":"e9df7298-84bf-4bec-b65b-6b034f106ed4","Type":"ContainerStarted","Data":"6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315"}
Oct 30 19:01:18.610036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:18.609241    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:18.638875 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:18.638843    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:18.640302 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:18.640265    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq" podUID="e9df7298-84bf-4bec-b65b-6b034f106ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 19:01:18.701186 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:18.701132    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq" podStartSLOduration=3.701119478 podStartE2EDuration="3.701119478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:18.700349128 +0000 UTC m=+1384.870705978" watchObservedRunningTime="2026-10-30 19:01:18.701119478 +0000 UTC m=+1384.871476304"
Oct 30 19:01:18.943745 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:18.943655    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:19.190421 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:19.190390    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:19.640745 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:19.640714    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:19.883863 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:01:19.883826    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:01:19.889238 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:19.889210    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:20.521018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:20.520987    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:20.596572 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:20.596542    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:20.597471 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:20.597452    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:23.944883 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:23.944846    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:29.278262 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:29.278204    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:29.608697 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:29.608670    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:32.733832 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:32.733801    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:33.049294 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:33.049199    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:33.265005 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:33.264972    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:37.967269 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:37.967233    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"]
Oct 30 19:01:37.967784 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:37.967480    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq" podUID="e9df7298-84bf-4bec-b65b-6b034f106ed4" containerName="istio-proxy" containerID="cri-o://6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315" gracePeriod=30
Oct 30 19:01:38.099377 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:38.099339    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:41.414378 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.414324    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:41.507587 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.507516    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-envoy\") pod \"e9df7298-84bf-4bec-b65b-6b034f106ed4\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") "
Oct 30 19:01:41.507587 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.507566    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-certs\") pod \"e9df7298-84bf-4bec-b65b-6b034f106ed4\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") "
Oct 30 19:01:41.507733 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.507590    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9df7298-84bf-4bec-b65b-6b034f106ed4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9df7298-84bf-4bec-b65b-6b034f106ed4-istiod-ca-cert\") pod \"e9df7298-84bf-4bec-b65b-6b034f106ed4\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") "
Oct 30 19:01:41.507733 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.507609    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-kube-api-access-gqkzf\" (UniqueName: \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-kube-api-access-gqkzf\") pod \"e9df7298-84bf-4bec-b65b-6b034f106ed4\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") "
Oct 30 19:01:41.507733 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.507644    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-token\" (UniqueName: \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-token\") pod \"e9df7298-84bf-4bec-b65b-6b034f106ed4\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") "
Oct 30 19:01:41.507733 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.507685    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-socket\") pod \"e9df7298-84bf-4bec-b65b-6b034f106ed4\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") "
Oct 30 19:01:41.507733 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.507711    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-credential-socket\") pod \"e9df7298-84bf-4bec-b65b-6b034f106ed4\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") "
Oct 30 19:01:41.507976 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.507754    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-podinfo\") pod \"e9df7298-84bf-4bec-b65b-6b034f106ed4\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") "
Oct 30 19:01:41.507976 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.507796    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-data\") pod \"e9df7298-84bf-4bec-b65b-6b034f106ed4\" (UID: \"e9df7298-84bf-4bec-b65b-6b034f106ed4\") "
Oct 30 19:01:41.507976 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.507919    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-certs" pod "e9df7298-84bf-4bec-b65b-6b034f106ed4" (UID: "e9df7298-84bf-4bec-b65b-6b034f106ed4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:41.508131 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.507997    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9df7298-84bf-4bec-b65b-6b034f106ed4-istiod-ca-cert" pod "e9df7298-84bf-4bec-b65b-6b034f106ed4" (UID: "e9df7298-84bf-4bec-b65b-6b034f106ed4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:41.508131 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.508063    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:41.508221 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.508122    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-data" pod "e9df7298-84bf-4bec-b65b-6b034f106ed4" (UID: "e9df7298-84bf-4bec-b65b-6b034f106ed4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:41.508221 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.508143    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-credential-socket" pod "e9df7298-84bf-4bec-b65b-6b034f106ed4" (UID: "e9df7298-84bf-4bec-b65b-6b034f106ed4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:41.508221 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.508160    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-socket" pod "e9df7298-84bf-4bec-b65b-6b034f106ed4" (UID: "e9df7298-84bf-4bec-b65b-6b034f106ed4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:41.509968 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.509930    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-envoy" pod "e9df7298-84bf-4bec-b65b-6b034f106ed4" (UID: "e9df7298-84bf-4bec-b65b-6b034f106ed4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:41.510085 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.510029    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-kube-api-access-gqkzf" pod "e9df7298-84bf-4bec-b65b-6b034f106ed4" (UID: "e9df7298-84bf-4bec-b65b-6b034f106ed4"). InnerVolumeSpecName "kube-api-access-gqkzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:41.510085 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.510062    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-token" pod "e9df7298-84bf-4bec-b65b-6b034f106ed4" (UID: "e9df7298-84bf-4bec-b65b-6b034f106ed4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:41.510225 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.510210    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-podinfo" pod "e9df7298-84bf-4bec-b65b-6b034f106ed4" (UID: "e9df7298-84bf-4bec-b65b-6b034f106ed4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:01:41.609255 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.609233    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:41.609360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.609259    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:41.609360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.609269    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9df7298-84bf-4bec-b65b-6b034f106ed4-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:41.609360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.609278    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqkzf\" (UniqueName: \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-kube-api-access-gqkzf\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:41.609360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.609287    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:41.609360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.609296    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:41.609360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.609305    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9df7298-84bf-4bec-b65b-6b034f106ed4-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:41.609360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.609313    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9df7298-84bf-4bec-b65b-6b034f106ed4-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:41.711011 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.710976    4853 generic.go:356] "Generic (PLEG): container finished" podID="e9df7298-84bf-4bec-b65b-6b034f106ed4" containerID="6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315" exitCode=0
Oct 30 19:01:41.711239 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.711096    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq" event={"ID":"e9df7298-84bf-4bec-b65b-6b034f106ed4","Type":"ContainerDied","Data":"6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315"}
Oct 30 19:01:41.711461 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.711166    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"
Oct 30 19:01:41.711574 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.711482    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq" event={"ID":"e9df7298-84bf-4bec-b65b-6b034f106ed4","Type":"ContainerDied","Data":"3df786d8e97cc225b6b1a4a8711b80c4a00619e813e522c60216a867183e5783"}
Oct 30 19:01:41.711574 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.711554    4853 scope.go:122] "RemoveContainer" containerID="6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315"
Oct 30 19:01:41.721368 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.721349    4853 scope.go:122] "RemoveContainer" containerID="6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315"
Oct 30 19:01:41.721646 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:01:41.721627    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315\": container with ID starting with 6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315 not found: ID does not exist" containerID="6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315"
Oct 30 19:01:41.721705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.721655    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315"} err="failed to get container status \"6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315\": rpc error: code = NotFound desc = could not find container \"6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315\": container with ID starting with 6c08f61a7ede39c50faff7c5b741e64742596700f6ab5aae8f2d233f7501d315 not found: ID does not exist"
Oct 30 19:01:41.870766 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.870732    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"]
Oct 30 19:01:41.872999 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.872981    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-zg4kq"]
Oct 30 19:01:41.939640 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:41.939617    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9df7298-84bf-4bec-b65b-6b034f106ed4" path="/var/lib/kubelet/pods/e9df7298-84bf-4bec-b65b-6b034f106ed4/volumes"
Oct 30 19:01:43.004320 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:43.004288    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:44.040314 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:44.040278    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:44.255400 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:44.255342    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:44.950199 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:44.950168    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:45.643932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:45.643899    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:49.537910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:49.537877    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:49.872653 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:01:49.872618    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:01:50.090556 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:50.089250    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:52.753022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:52.752982    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:54.840735 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:54.840697    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:56.117506 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:56.117470    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"]
Oct 30 19:01:56.127569 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:01:56.127517    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f" cmd=["/bin/sleep","15"]
Oct 30 19:01:56.127711 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:56.127597    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:01:56.127711 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:01:56.127611    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" podUID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerName="main" containerID="cri-o://f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f"
Oct 30 19:01:56.127711 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:56.127648    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" podUID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerName="main" containerID="cri-o://f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f" gracePeriod=60
Oct 30 19:01:56.156974 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:56.156942    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:56.390986 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:56.390899    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:56.589308 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:56.589272    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" podUID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8001/health\": dial tcp 10.134.0.41:8001: connect: connection refused"
Oct 30 19:01:56.591497 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:56.591464    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:56.740090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:56.740063    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" podUID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerName="llm-d-routing-sidecar" containerID="cri-o://e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161" gracePeriod=60
Oct 30 19:01:56.789661 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:56.789637    4853 generic.go:356] "Generic (PLEG): container finished" podID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerID="f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f" exitCode=0
Oct 30 19:01:56.789756 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:56.789688    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" event={"ID":"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0","Type":"ContainerDied","Data":"f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f"}
Oct 30 19:01:56.983310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:56.983284    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:57.144420 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.144397    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-kube-api-access-k589n\" (UniqueName: \"kubernetes.io/projected/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-kube-api-access-k589n\") pod \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") "
Oct 30 19:01:57.144896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.144446    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-dshm\") pod \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") "
Oct 30 19:01:57.144896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.144468    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tmp-dir\") pod \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") "
Oct 30 19:01:57.144896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.144513    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-model-cache\") pod \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") "
Oct 30 19:01:57.144896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.144578    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-home\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-home\") pod \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") "
Oct 30 19:01:57.144896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.144626    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tls-certs\") pod \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\" (UID: \"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0\") "
Oct 30 19:01:57.144896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.144823    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-model-cache" pod "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" (UID: "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:57.144896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.144859    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-home" pod "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" (UID: "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:57.144896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.144831    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tmp-dir" pod "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" (UID: "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:57.146815 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.146789    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-dshm" pod "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" (UID: "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:57.146815 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.146800    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-kube-api-access-k589n" pod "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" (UID: "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0"). InnerVolumeSpecName "kube-api-access-k589n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:57.146815 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.146799    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tls-certs" pod "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" (UID: "dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:57.246051 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.246026    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:57.246051 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.246049    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:57.246185 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.246059    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:57.246185 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.246069    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:57.246185 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.246078    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:57.246185 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.246087    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k589n\" (UniqueName: \"kubernetes.io/projected/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0-kube-api-access-k589n\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:57.796080 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.796048    4853 generic.go:356] "Generic (PLEG): container finished" podID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerID="e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161" exitCode=0
Oct 30 19:01:57.796244 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.796111    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"
Oct 30 19:01:57.796244 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.796144    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" event={"ID":"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0","Type":"ContainerDied","Data":"e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161"}
Oct 30 19:01:57.796244 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.796231    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m" event={"ID":"dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0","Type":"ContainerDied","Data":"09cb75fe3b3bb1e37330d913bf6ac5c79f3cb52be7a20957b62c0e276fc63722"}
Oct 30 19:01:57.796361 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.796258    4853 scope.go:122] "RemoveContainer" containerID="f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f"
Oct 30 19:01:57.806105 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.806089    4853 scope.go:122] "RemoveContainer" containerID="e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161"
Oct 30 19:01:57.813377 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.813355    4853 scope.go:122] "RemoveContainer" containerID="f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f"
Oct 30 19:01:57.813670 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:01:57.813648    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f\": container with ID starting with f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f not found: ID does not exist" containerID="f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f"
Oct 30 19:01:57.813737 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.813683    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f"} err="failed to get container status \"f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f\": rpc error: code = NotFound desc = could not find container \"f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f\": container with ID starting with f9f206319c8fca9673bebd59cbfc8f7cb09fde61646e88852b44e9533b9dbf1f not found: ID does not exist"
Oct 30 19:01:57.813737 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.813707    4853 scope.go:122] "RemoveContainer" containerID="e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161"
Oct 30 19:01:57.814015 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:01:57.813995    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161\": container with ID starting with e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161 not found: ID does not exist" containerID="e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161"
Oct 30 19:01:57.814069 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.814023    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161"} err="failed to get container status \"e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161\": rpc error: code = NotFound desc = could not find container \"e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161\": container with ID starting with e1ff71afe6343f8a3e0727f45dff1ddc3523cf1d9c1cc0c755b0821c0de6b161 not found: ID does not exist"
Oct 30 19:01:57.953065 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.953043    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"]
Oct 30 19:01:57.999655 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:57.999627    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-rp96m"]
Oct 30 19:01:58.501299 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:58.501264    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:59.500516 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:59.500480    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:59.699172 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:59.699145    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:59.938278 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:01:59.938248    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" path="/var/lib/kubelet/pods/dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0/volumes"
Oct 30 19:02:09.487693 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:09.487655    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:12.134936 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:12.134851    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:12.359854 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:12.359800    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:19.876397 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:02:19.876355    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:02:44.110091 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:44.110055    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:44.370995 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:44.370884    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:49.872544 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:02:49.872494    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:02:51.700851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.700812    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"]
Oct 30 19:02:51.701268 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.701151    4853 cpu_manager.go:414] "containerMap: removing container" podUID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerName="main"
Oct 30 19:02:51.701268 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.701162    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerName="main"
Oct 30 19:02:51.701268 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.701188    4853 cpu_manager.go:414] "containerMap: removing container" podUID="e9df7298-84bf-4bec-b65b-6b034f106ed4" containerName="istio-proxy"
Oct 30 19:02:51.701268 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.701193    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9df7298-84bf-4bec-b65b-6b034f106ed4" containerName="istio-proxy"
Oct 30 19:02:51.701268 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.701207    4853 cpu_manager.go:414] "containerMap: removing container" podUID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerName="llm-d-routing-sidecar"
Oct 30 19:02:51.701268 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.701213    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerName="llm-d-routing-sidecar"
Oct 30 19:02:51.701268 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.701265    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerName="main"
Oct 30 19:02:51.701486 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.701276    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9df7298-84bf-4bec-b65b-6b034f106ed4" containerName="istio-proxy"
Oct 30 19:02:51.701486 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.701282    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce7050b-1bdb-4d5c-b74c-33f7ac2a00c0" containerName="llm-d-routing-sidecar"
Oct 30 19:02:51.704406 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.704383    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.745649 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.745625    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shr5t\" (UniqueName: \"kubernetes.io/projected/3a647fbe-9131-4cde-b844-f4fd74379752-kube-api-access-shr5t\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.745764 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.745659    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-model-cache\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.745764 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.745690    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.745764 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.745718    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-home\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.745882 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.745772    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-dshm\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.745882 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.745843    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a647fbe-9131-4cde-b844-f4fd74379752-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.745981 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.745956    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.746037 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.746020    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a647fbe-9131-4cde-b844-f4fd74379752-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.767495 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.767473    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Oct 30 19:02:51.807512 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.807491    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-j2j9z\""
Oct 30 19:02:51.807637 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.807511    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Oct 30 19:02:51.809461 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.809442    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"]
Oct 30 19:02:51.846439 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.846415    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a647fbe-9131-4cde-b844-f4fd74379752-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.846562 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.846458    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shr5t\" (UniqueName: \"kubernetes.io/projected/3a647fbe-9131-4cde-b844-f4fd74379752-kube-api-access-shr5t\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.846562 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.846479    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-model-cache\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.846562 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.846509    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.846562 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.846556    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-home\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.846758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.846570    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-dshm\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.846758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.846588    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a647fbe-9131-4cde-b844-f4fd74379752-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.846758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.846613    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.846945 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.846923    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.847006 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.846964    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-model-cache\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.847075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.847002    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-home\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.847121 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.847083    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.849009 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.848987    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-dshm\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.849445 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.849230    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a647fbe-9131-4cde-b844-f4fd74379752-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.854163 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.854146    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Oct 30 19:02:51.854286 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.854166    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:02:51.854286 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.854210    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Oct 30 19:02:51.857733 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.857713    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a647fbe-9131-4cde-b844-f4fd74379752-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:51.869723 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.869698    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:51.905628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:51.905595    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shr5t\" (UniqueName: \"kubernetes.io/projected/3a647fbe-9131-4cde-b844-f4fd74379752-kube-api-access-shr5t\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:52.019409 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:52.019342    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:02:52.242777 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:52.242749    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:52.449475 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:52.449453    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"]
Oct 30 19:02:52.457850 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:02:52.457823    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a647fbe_9131_4cde_b844_f4fd74379752.slice/crio-5076fbb4304d46704f0850108b3b4c828c6fd7b360fc52d689497c6b40af9f91 WatchSource:0}: Error finding container 5076fbb4304d46704f0850108b3b4c828c6fd7b360fc52d689497c6b40af9f91: Status 404 returned error can't find the container with id 5076fbb4304d46704f0850108b3b4c828c6fd7b360fc52d689497c6b40af9f91
Oct 30 19:02:52.621101 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:52.621052    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:52.860291 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:52.860252    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:53.062790 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:53.062755    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm" event={"ID":"3a647fbe-9131-4cde-b844-f4fd74379752","Type":"ContainerStarted","Data":"5076fbb4304d46704f0850108b3b4c828c6fd7b360fc52d689497c6b40af9f91"}
Oct 30 19:02:56.878885 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:56.878851    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:57.086596 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:57.086549    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm" event={"ID":"3a647fbe-9131-4cde-b844-f4fd74379752","Type":"ContainerStarted","Data":"94146d50388d5ac0cf23ab7d0536a45526ac161eb2da923288980931b4641788"}
Oct 30 19:02:59.074067 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:59.074027    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:59.377275 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:02:59.377247    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:00.101564 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:00.101513    4853 generic.go:356] "Generic (PLEG): container finished" podID="3a647fbe-9131-4cde-b844-f4fd74379752" containerID="94146d50388d5ac0cf23ab7d0536a45526ac161eb2da923288980931b4641788" exitCode=0
Oct 30 19:03:00.101925 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:00.101584    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm" event={"ID":"3a647fbe-9131-4cde-b844-f4fd74379752","Type":"ContainerDied","Data":"94146d50388d5ac0cf23ab7d0536a45526ac161eb2da923288980931b4641788"}
Oct 30 19:03:00.356367 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:00.356243    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:01.108312 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:01.108280    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm" event={"ID":"3a647fbe-9131-4cde-b844-f4fd74379752","Type":"ContainerStarted","Data":"74a94da7eded493da8bc19431b0d7a201b628162c0c7a3e772dc94573c7025ce"}
Oct 30 19:03:01.275890 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:01.275841    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm" podStartSLOduration=3.548618593 podStartE2EDuration="10.275828172s" totalImagesPullingTime="4.534868975s" totalInitContainerRuntime="2.192340604s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:01.275362269 +0000 UTC m=+1487.445719116" watchObservedRunningTime="2026-10-30 19:03:01.275828172 +0000 UTC m=+1487.446185019"
Oct 30 19:03:02.020446 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:02.020416    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:03:02.020628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:02.020470    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:03:02.033340 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:02.033308    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:03:02.124244 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:02.124222    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:03:19.872725 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:03:19.872690    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:03:23.444785 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:23.444736    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:36.003338 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.003308    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"]
Oct 30 19:03:36.013208 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:03:36.013174    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="74a94da7eded493da8bc19431b0d7a201b628162c0c7a3e772dc94573c7025ce" cmd=["/bin/sleep","15"]
Oct 30 19:03:36.013350 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.013226    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:03:36.013350 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:03:36.013238    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm" podUID="3a647fbe-9131-4cde-b844-f4fd74379752" containerName="main" containerID="cri-o://74a94da7eded493da8bc19431b0d7a201b628162c0c7a3e772dc94573c7025ce"
Oct 30 19:03:36.013350 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.013283    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm" podUID="3a647fbe-9131-4cde-b844-f4fd74379752" containerName="main" containerID="cri-o://74a94da7eded493da8bc19431b0d7a201b628162c0c7a3e772dc94573c7025ce" gracePeriod=60
Oct 30 19:03:36.057906 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.057883    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:36.113488 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.113438    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm" podUID="3a647fbe-9131-4cde-b844-f4fd74379752" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Oct 30 19:03:36.283524 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.283485    4853 generic.go:356] "Generic (PLEG): container finished" podID="3a647fbe-9131-4cde-b844-f4fd74379752" containerID="74a94da7eded493da8bc19431b0d7a201b628162c0c7a3e772dc94573c7025ce" exitCode=0
Oct 30 19:03:36.283524 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.283513    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm" event={"ID":"3a647fbe-9131-4cde-b844-f4fd74379752","Type":"ContainerDied","Data":"74a94da7eded493da8bc19431b0d7a201b628162c0c7a3e772dc94573c7025ce"}
Oct 30 19:03:36.349886 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.349826    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:36.364684 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.364656    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:03:36.482240 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482211    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-home\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-home\") pod \"3a647fbe-9131-4cde-b844-f4fd74379752\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") "
Oct 30 19:03:36.482387 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482260    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-dshm\") pod \"3a647fbe-9131-4cde-b844-f4fd74379752\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") "
Oct 30 19:03:36.482387 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482292    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a647fbe-9131-4cde-b844-f4fd74379752-kube-api-access-shr5t\" (UniqueName: \"kubernetes.io/projected/3a647fbe-9131-4cde-b844-f4fd74379752-kube-api-access-shr5t\") pod \"3a647fbe-9131-4cde-b844-f4fd74379752\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") "
Oct 30 19:03:36.482387 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482328    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-tmp-dir\") pod \"3a647fbe-9131-4cde-b844-f4fd74379752\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") "
Oct 30 19:03:36.482387 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482373    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a647fbe-9131-4cde-b844-f4fd74379752-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a647fbe-9131-4cde-b844-f4fd74379752-tls-certs\") pod \"3a647fbe-9131-4cde-b844-f4fd74379752\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") "
Oct 30 19:03:36.482617 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482400    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a647fbe-9131-4cde-b844-f4fd74379752-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a647fbe-9131-4cde-b844-f4fd74379752-cabundle-cert\") pod \"3a647fbe-9131-4cde-b844-f4fd74379752\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") "
Oct 30 19:03:36.482617 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482460    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-kserve-provision-location\") pod \"3a647fbe-9131-4cde-b844-f4fd74379752\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") "
Oct 30 19:03:36.482617 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482494    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-model-cache\") pod \"3a647fbe-9131-4cde-b844-f4fd74379752\" (UID: \"3a647fbe-9131-4cde-b844-f4fd74379752\") "
Oct 30 19:03:36.482617 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482511    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-home" pod "3a647fbe-9131-4cde-b844-f4fd74379752" (UID: "3a647fbe-9131-4cde-b844-f4fd74379752"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.482821 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482676    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-tmp-dir" pod "3a647fbe-9131-4cde-b844-f4fd74379752" (UID: "3a647fbe-9131-4cde-b844-f4fd74379752"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.482881 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482835    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.482881 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482852    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.482881 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482852    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-model-cache" pod "3a647fbe-9131-4cde-b844-f4fd74379752" (UID: "3a647fbe-9131-4cde-b844-f4fd74379752"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.482881 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.482869    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a647fbe-9131-4cde-b844-f4fd74379752-cabundle-cert" pod "3a647fbe-9131-4cde-b844-f4fd74379752" (UID: "3a647fbe-9131-4cde-b844-f4fd74379752"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:36.485343 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.485321    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-dshm" pod "3a647fbe-9131-4cde-b844-f4fd74379752" (UID: "3a647fbe-9131-4cde-b844-f4fd74379752"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.485470 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.485370    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a647fbe-9131-4cde-b844-f4fd74379752-tls-certs" pod "3a647fbe-9131-4cde-b844-f4fd74379752" (UID: "3a647fbe-9131-4cde-b844-f4fd74379752"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:36.485470 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.485423    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a647fbe-9131-4cde-b844-f4fd74379752-kube-api-access-shr5t" pod "3a647fbe-9131-4cde-b844-f4fd74379752" (UID: "3a647fbe-9131-4cde-b844-f4fd74379752"). InnerVolumeSpecName "kube-api-access-shr5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:36.502866 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.502846    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-kserve-provision-location" pod "3a647fbe-9131-4cde-b844-f4fd74379752" (UID: "3a647fbe-9131-4cde-b844-f4fd74379752"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.583871 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.583840    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.584000 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.583874    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.584000 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.583888    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a647fbe-9131-4cde-b844-f4fd74379752-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.584000 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.583901    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shr5t\" (UniqueName: \"kubernetes.io/projected/3a647fbe-9131-4cde-b844-f4fd74379752-kube-api-access-shr5t\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.584000 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.583913    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a647fbe-9131-4cde-b844-f4fd74379752-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.584000 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:36.583926    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a647fbe-9131-4cde-b844-f4fd74379752-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:37.290311 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:37.290235    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"
Oct 30 19:03:37.290311 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:37.290262    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm" event={"ID":"3a647fbe-9131-4cde-b844-f4fd74379752","Type":"ContainerDied","Data":"5076fbb4304d46704f0850108b3b4c828c6fd7b360fc52d689497c6b40af9f91"}
Oct 30 19:03:37.290947 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:37.290320    4853 scope.go:122] "RemoveContainer" containerID="74a94da7eded493da8bc19431b0d7a201b628162c0c7a3e772dc94573c7025ce"
Oct 30 19:03:37.300355 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:37.300337    4853 scope.go:122] "RemoveContainer" containerID="94146d50388d5ac0cf23ab7d0536a45526ac161eb2da923288980931b4641788"
Oct 30 19:03:37.501902 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:37.501876    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"]
Oct 30 19:03:37.549384 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:37.549324    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-dqqvm"]
Oct 30 19:03:37.939291 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:37.939265    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a647fbe-9131-4cde-b844-f4fd74379752" path="/var/lib/kubelet/pods/3a647fbe-9131-4cde-b844-f4fd74379752/volumes"
Oct 30 19:03:42.270499 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:42.270462    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:42.530436 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:42.530348    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:49.874940 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:03:49.874901    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:03:57.261335 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:03:57.261293    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:00.159392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:00.159345    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:00.369552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:00.369500    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:19.875959 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:04:19.875914    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:04:23.680363 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:23.680327    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:24.002616 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:24.002521    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:32.070443 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:32.070413    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:34.191409 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.191373    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"]
Oct 30 19:04:34.191826 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.191752    4853 cpu_manager.go:414] "containerMap: removing container" podUID="3a647fbe-9131-4cde-b844-f4fd74379752" containerName="main"
Oct 30 19:04:34.191826 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.191765    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a647fbe-9131-4cde-b844-f4fd74379752" containerName="main"
Oct 30 19:04:34.191826 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.191787    4853 cpu_manager.go:414] "containerMap: removing container" podUID="3a647fbe-9131-4cde-b844-f4fd74379752" containerName="storage-initializer"
Oct 30 19:04:34.191826 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.191793    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a647fbe-9131-4cde-b844-f4fd74379752" containerName="storage-initializer"
Oct 30 19:04:34.191958 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.191855    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a647fbe-9131-4cde-b844-f4fd74379752" containerName="main"
Oct 30 19:04:34.194851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.194828    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.245250 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.245222    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Oct 30 19:04:34.257164 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.257138    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Oct 30 19:04:34.290942 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.290909    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:04:34.291075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.290951    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-kbdpg\""
Oct 30 19:04:34.291075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.290961    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Oct 30 19:04:34.291075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.291018    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Oct 30 19:04:34.293643 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.293620    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"]
Oct 30 19:04:34.298215 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.298194    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.298345 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.298221    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.298345 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.298242    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtfgg\" (UniqueName: \"kubernetes.io/projected/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kube-api-access-jtfgg\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.298345 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.298268    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.298551 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.298406    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-model-cache\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.298551 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.298446    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-home\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.298551 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.298470    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-dshm\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.298724 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.298580    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.345577 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.345547    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:34.399325 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.399294    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.399442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.399334    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-model-cache\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.399442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.399358    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-home\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.399442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.399374    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-dshm\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.399442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.399408    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.399442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.399439    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.399793 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.399455    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.399793 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.399501    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfgg\" (UniqueName: \"kubernetes.io/projected/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kube-api-access-jtfgg\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.399896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.399799    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-model-cache\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.399896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.399820    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-home\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.399896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.399886    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.400081 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.400063    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.400214 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.400196    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.401873 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.401854    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-dshm\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.401961 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.401947    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.454224 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.454156    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfgg\" (UniqueName: \"kubernetes.io/projected/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kube-api-access-jtfgg\") pod \"tokenizer-migration-path-test-kserve-75768b77f-t6vgc\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.510262 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.510233    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:34.934592 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:34.934558    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"]
Oct 30 19:04:34.941684 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:04:34.941637    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6724a4e5_c8cc_45e9_a41d_4e9985c36c92.slice/crio-253c8298d504673c259bd0ca289cd4bf8b1df40312393fbd315268be4d3b6a70 WatchSource:0}: Error finding container 253c8298d504673c259bd0ca289cd4bf8b1df40312393fbd315268be4d3b6a70: Status 404 returned error can't find the container with id 253c8298d504673c259bd0ca289cd4bf8b1df40312393fbd315268be4d3b6a70
Oct 30 19:04:35.010791 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:35.010758    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:35.242196 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:35.242105    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:35.570413 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:35.570332    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc" event={"ID":"6724a4e5-c8cc-45e9-a41d-4e9985c36c92","Type":"ContainerStarted","Data":"534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35"}
Oct 30 19:04:35.570413 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:35.570399    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc" event={"ID":"6724a4e5-c8cc-45e9-a41d-4e9985c36c92","Type":"ContainerStarted","Data":"253c8298d504673c259bd0ca289cd4bf8b1df40312393fbd315268be4d3b6a70"}
Oct 30 19:04:37.584678 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:37.584648    4853 generic.go:356] "Generic (PLEG): container finished" podID="6724a4e5-c8cc-45e9-a41d-4e9985c36c92" containerID="534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35" exitCode=0
Oct 30 19:04:37.585023 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:37.584717    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc" event={"ID":"6724a4e5-c8cc-45e9-a41d-4e9985c36c92","Type":"ContainerDied","Data":"534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35"}
Oct 30 19:04:38.590859 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:38.590823    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc" event={"ID":"6724a4e5-c8cc-45e9-a41d-4e9985c36c92","Type":"ContainerStarted","Data":"04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a"}
Oct 30 19:04:38.756238 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:38.756176    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc" podStartSLOduration=2.625887035 podStartE2EDuration="4.756156955s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.13026992s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:38.753183111 +0000 UTC m=+1584.923539960" watchObservedRunningTime="2026-10-30 19:04:38.756156955 +0000 UTC m=+1584.926513804"
Oct 30 19:04:39.241325 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:39.241277    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:39.510882 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:39.510820    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:39.553243 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:39.553215    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:41.902670 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:41.902613    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:44.510785 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:44.510751    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:44.522984 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:44.522961    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:44.628306 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:44.628280    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:04:48.622170 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:48.622133    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:48.906334 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:48.906191    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:49.875457 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:04:49.875423    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:04:56.586029 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:56.585999    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:57.615745 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.615676    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:57.695517 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.695492    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"]
Oct 30 19:04:57.698444 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.698423    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.755770 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.755746    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Oct 30 19:04:57.755770 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.755746    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Oct 30 19:04:57.755960 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.755794    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:04:57.755960 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.755864    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-lzg96\""
Oct 30 19:04:57.755960 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.755872    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Oct 30 19:04:57.755960 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.755914    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Oct 30 19:04:57.784947 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.784919    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-model-cache\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.785049 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.784947    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz6sd\" (UniqueName: \"kubernetes.io/projected/0272eff8-6572-4933-8cbe-a6db1d967ad3-kube-api-access-rz6sd\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.785049 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.785009    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0272eff8-6572-4933-8cbe-a6db1d967ad3-tls-certs\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.785049 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.785046    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0272eff8-6572-4933-8cbe-a6db1d967ad3-cabundle-cert\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.785155 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.785067    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-home\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.785188 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.785163    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-dshm\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.785267 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.785251    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-tmp-dir\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.785302 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.785284    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.804448 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.804425    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"]
Oct 30 19:04:57.886043 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.885985    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0272eff8-6572-4933-8cbe-a6db1d967ad3-cabundle-cert\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886043 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886016    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-home\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886182 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886046    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-dshm\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886182 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886089    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-tmp-dir\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886182 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886118    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886182 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886159    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-model-cache\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886182 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886180    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz6sd\" (UniqueName: \"kubernetes.io/projected/0272eff8-6572-4933-8cbe-a6db1d967ad3-kube-api-access-rz6sd\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886433 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886218    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0272eff8-6572-4933-8cbe-a6db1d967ad3-tls-certs\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886491 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886440    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-home\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886618 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886600    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-tmp-dir\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886680 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886641    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-model-cache\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886717 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886688    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.886782 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.886766    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0272eff8-6572-4933-8cbe-a6db1d967ad3-cabundle-cert\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.888635 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.888614    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-dshm\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.888911 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.888897    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0272eff8-6572-4933-8cbe-a6db1d967ad3-tls-certs\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:57.897929 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.897891    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:57.942603 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:57.942572    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz6sd\" (UniqueName: \"kubernetes.io/projected/0272eff8-6572-4933-8cbe-a6db1d967ad3-kube-api-access-rz6sd\") pod \"auth-invalid-token-test-kserve-54976cfdb7-5lsrv\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:58.013642 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:58.013615    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:04:58.513306 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:58.513272    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"]
Oct 30 19:04:58.541795 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:04:58.541769    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0272eff8_6572_4933_8cbe_a6db1d967ad3.slice/crio-a5c082ec318b094503e9edc73b278e6786b3a8894edd3e9a606221303e724b9d WatchSource:0}: Error finding container a5c082ec318b094503e9edc73b278e6786b3a8894edd3e9a606221303e724b9d: Status 404 returned error can't find the container with id a5c082ec318b094503e9edc73b278e6786b3a8894edd3e9a606221303e724b9d
Oct 30 19:04:58.563662 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:58.563634    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:58.684472 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:58.684429    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" event={"ID":"0272eff8-6572-4933-8cbe-a6db1d967ad3","Type":"ContainerStarted","Data":"6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11"}
Oct 30 19:04:58.685209 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:04:58.684559    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" event={"ID":"0272eff8-6572-4933-8cbe-a6db1d967ad3","Type":"ContainerStarted","Data":"a5c082ec318b094503e9edc73b278e6786b3a8894edd3e9a606221303e724b9d"}
Oct 30 19:05:01.700178 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:01.700136    4853 generic.go:356] "Generic (PLEG): container finished" podID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerID="6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11" exitCode=0
Oct 30 19:05:01.700687 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:01.700185    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" event={"ID":"0272eff8-6572-4933-8cbe-a6db1d967ad3","Type":"ContainerDied","Data":"6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11"}
Oct 30 19:05:03.191579 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:03.191545    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:03.578690 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:03.578611    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:05.337936 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:05.337860    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:18.142856 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.142821    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"]
Oct 30 19:05:18.152595 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:05:18.152549    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a" cmd=["/bin/sleep","15"]
Oct 30 19:05:18.152723 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.152604    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:05:18.152723 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:05:18.152616    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc" podUID="6724a4e5-c8cc-45e9-a41d-4e9985c36c92" containerName="main" containerID="cri-o://04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a"
Oct 30 19:05:18.152723 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.152666    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc" podUID="6724a4e5-c8cc-45e9-a41d-4e9985c36c92" containerName="main" containerID="cri-o://04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a" gracePeriod=60
Oct 30 19:05:18.261251 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.261223    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:18.500493 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.500460    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:18.565766 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.565740    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:05:18.696138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.696069    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-cabundle-cert\") pod \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") "
Oct 30 19:05:18.696138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.696108    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tmp-dir\") pod \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") "
Oct 30 19:05:18.696138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.696137    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kube-api-access-jtfgg\" (UniqueName: \"kubernetes.io/projected/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kube-api-access-jtfgg\") pod \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") "
Oct 30 19:05:18.696343 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.696160    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tls-certs\" (UniqueName: \"kubernetes.io/secret/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tls-certs\") pod \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") "
Oct 30 19:05:18.696343 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.696177    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-dshm\") pod \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") "
Oct 30 19:05:18.696343 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.696239    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-model-cache\") pod \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") "
Oct 30 19:05:18.696343 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.696262    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kserve-provision-location\") pod \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") "
Oct 30 19:05:18.696343 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.696299    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-home\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-home\") pod \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\" (UID: \"6724a4e5-c8cc-45e9-a41d-4e9985c36c92\") "
Oct 30 19:05:18.699201 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.696655    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-home" pod "6724a4e5-c8cc-45e9-a41d-4e9985c36c92" (UID: "6724a4e5-c8cc-45e9-a41d-4e9985c36c92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.699201 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.696807    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tmp-dir" pod "6724a4e5-c8cc-45e9-a41d-4e9985c36c92" (UID: "6724a4e5-c8cc-45e9-a41d-4e9985c36c92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.699201 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.697013    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-cabundle-cert" pod "6724a4e5-c8cc-45e9-a41d-4e9985c36c92" (UID: "6724a4e5-c8cc-45e9-a41d-4e9985c36c92"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:18.699201 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.697113    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-model-cache" pod "6724a4e5-c8cc-45e9-a41d-4e9985c36c92" (UID: "6724a4e5-c8cc-45e9-a41d-4e9985c36c92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.699946 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.699879    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-dshm" pod "6724a4e5-c8cc-45e9-a41d-4e9985c36c92" (UID: "6724a4e5-c8cc-45e9-a41d-4e9985c36c92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.700022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.699967    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kube-api-access-jtfgg" pod "6724a4e5-c8cc-45e9-a41d-4e9985c36c92" (UID: "6724a4e5-c8cc-45e9-a41d-4e9985c36c92"). InnerVolumeSpecName "kube-api-access-jtfgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:18.701010 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.700985    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tls-certs" pod "6724a4e5-c8cc-45e9-a41d-4e9985c36c92" (UID: "6724a4e5-c8cc-45e9-a41d-4e9985c36c92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:18.719762 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.719727    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kserve-provision-location" pod "6724a4e5-c8cc-45e9-a41d-4e9985c36c92" (UID: "6724a4e5-c8cc-45e9-a41d-4e9985c36c92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.797435 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.797405    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.797435 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.797438    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.797641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.797451    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtfgg\" (UniqueName: \"kubernetes.io/projected/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kube-api-access-jtfgg\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.797641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.797462    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.797641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.797472    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.797641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.797479    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.797641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.797488    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.797641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.797496    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6724a4e5-c8cc-45e9-a41d-4e9985c36c92-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.824759 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.824720    4853 generic.go:356] "Generic (PLEG): container finished" podID="6724a4e5-c8cc-45e9-a41d-4e9985c36c92" containerID="04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a" exitCode=0
Oct 30 19:05:18.824919 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.824780    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc" event={"ID":"6724a4e5-c8cc-45e9-a41d-4e9985c36c92","Type":"ContainerDied","Data":"04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a"}
Oct 30 19:05:18.824919 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.824807    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"
Oct 30 19:05:18.824919 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.824824    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc" event={"ID":"6724a4e5-c8cc-45e9-a41d-4e9985c36c92","Type":"ContainerDied","Data":"253c8298d504673c259bd0ca289cd4bf8b1df40312393fbd315268be4d3b6a70"}
Oct 30 19:05:18.824919 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.824845    4853 scope.go:122] "RemoveContainer" containerID="04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a"
Oct 30 19:05:18.835848 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.835761    4853 scope.go:122] "RemoveContainer" containerID="534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35"
Oct 30 19:05:18.844976 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.844957    4853 scope.go:122] "RemoveContainer" containerID="04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a"
Oct 30 19:05:18.845289 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:05:18.845270    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a\": container with ID starting with 04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a not found: ID does not exist" containerID="04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a"
Oct 30 19:05:18.845359 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.845297    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a"} err="failed to get container status \"04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a\": rpc error: code = NotFound desc = could not find container \"04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a\": container with ID starting with 04d010895018a6e6d2611a6f4769658d4d79af1ea87817ea3e75fddf6cad638a not found: ID does not exist"
Oct 30 19:05:18.845359 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.845316    4853 scope.go:122] "RemoveContainer" containerID="534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35"
Oct 30 19:05:18.845575 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:05:18.845558    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35\": container with ID starting with 534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35 not found: ID does not exist" containerID="534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35"
Oct 30 19:05:18.845620 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.845580    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35"} err="failed to get container status \"534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35\": rpc error: code = NotFound desc = could not find container \"534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35\": container with ID starting with 534ce830cca84016a4386061e98824e795d1626681feca3c7bcbfd91a777aa35 not found: ID does not exist"
Oct 30 19:05:18.987170 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.987095    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"]
Oct 30 19:05:18.995774 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:18.995746    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-t6vgc"]
Oct 30 19:05:19.875811 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:05:19.875776    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:05:19.939937 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:19.939904    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6724a4e5-c8cc-45e9-a41d-4e9985c36c92" path="/var/lib/kubelet/pods/6724a4e5-c8cc-45e9-a41d-4e9985c36c92/volumes"
Oct 30 19:05:31.899502 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:31.899464    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" event={"ID":"0272eff8-6572-4933-8cbe-a6db1d967ad3","Type":"ContainerStarted","Data":"e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b"}
Oct 30 19:05:32.014486 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:32.014447    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:05:32.118430 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:32.118377    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podStartSLOduration=3.10473071 podStartE2EDuration="35.118363853s" totalImagesPullingTime="29.693387814s" totalInitContainerRuntime="2.320245329s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:32.070752426 +0000 UTC m=+1638.241109276" watchObservedRunningTime="2026-10-30 19:05:32.118363853 +0000 UTC m=+1638.288720705"
Oct 30 19:05:38.014382 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:38.014346    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:05:38.015798 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:38.015775    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Oct 30 19:05:48.014868 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:48.014829    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Oct 30 19:05:49.873782 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:05:49.873744    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:05:57.907206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:57.907121    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:58.014913 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:05:58.014868    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Oct 30 19:06:05.394904 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:05.394865    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:05.654810 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:05.654732    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:08.014431 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:08.014390    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Oct 30 19:06:12.903368 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:12.903325    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:14.447427 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.447385    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"]
Oct 30 19:06:14.447945 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.447825    4853 cpu_manager.go:414] "containerMap: removing container" podUID="6724a4e5-c8cc-45e9-a41d-4e9985c36c92" containerName="main"
Oct 30 19:06:14.447945 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.447842    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6724a4e5-c8cc-45e9-a41d-4e9985c36c92" containerName="main"
Oct 30 19:06:14.447945 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.447863    4853 cpu_manager.go:414] "containerMap: removing container" podUID="6724a4e5-c8cc-45e9-a41d-4e9985c36c92" containerName="storage-initializer"
Oct 30 19:06:14.447945 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.447868    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6724a4e5-c8cc-45e9-a41d-4e9985c36c92" containerName="storage-initializer"
Oct 30 19:06:14.448134 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.447950    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="6724a4e5-c8cc-45e9-a41d-4e9985c36c92" containerName="main"
Oct 30 19:06:14.463052 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.462974    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.474609 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.474587    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-home\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.474733 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.474628    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-model-cache\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.474775 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.474750    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.474837 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.474822    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmwks\" (UniqueName: \"kubernetes.io/projected/2e3e96d1-8201-411c-8e71-132718d75452-kube-api-access-nmwks\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.474876 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.474864    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-dshm\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.474919 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.474893    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3e96d1-8201-411c-8e71-132718d75452-tls-certs\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.474954 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.474919    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.474992 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.474948    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e3e96d1-8201-411c-8e71-132718d75452-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.515068 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.515031    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Oct 30 19:06:14.561551 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.561509    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Oct 30 19:06:14.561786 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.561759    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-5mrjf\""
Oct 30 19:06:14.561968 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.561776    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:06:14.562074 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.561792    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Oct 30 19:06:14.562074 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.561851    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Oct 30 19:06:14.563657 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.563639    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"]
Oct 30 19:06:14.575932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.575899    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmwks\" (UniqueName: \"kubernetes.io/projected/2e3e96d1-8201-411c-8e71-132718d75452-kube-api-access-nmwks\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576028 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.575954    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-dshm\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576028 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.575973    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3e96d1-8201-411c-8e71-132718d75452-tls-certs\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576028 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.575989    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576028 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.576006    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e3e96d1-8201-411c-8e71-132718d75452-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576221 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.576050    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-home\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576264 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.576217    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-model-cache\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.576297    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.576420    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-home\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576620 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.576445    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576755 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.576735    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-model-cache\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576835 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.576788    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.576835 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.576816    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e3e96d1-8201-411c-8e71-132718d75452-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.578572 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.578512    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-dshm\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.579149 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.579126    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3e96d1-8201-411c-8e71-132718d75452-tls-certs\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.590593 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.590522    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:14.633118 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.633094    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmwks\" (UniqueName: \"kubernetes.io/projected/2e3e96d1-8201-411c-8e71-132718d75452-kube-api-access-nmwks\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-4x5hx\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:14.780895 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:14.780806    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:15.234696 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:15.231839    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"]
Oct 30 19:06:15.240003 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:15.239969    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:15.240202 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:06:15.240009    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3e96d1_8201_411c_8e71_132718d75452.slice/crio-c470587cac4973ee59b21837e29cbce2b0ae38b8f32375c177e86734c4fc4194 WatchSource:0}: Error finding container c470587cac4973ee59b21837e29cbce2b0ae38b8f32375c177e86734c4fc4194: Status 404 returned error can't find the container with id c470587cac4973ee59b21837e29cbce2b0ae38b8f32375c177e86734c4fc4194
Oct 30 19:06:15.450961 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:15.450918    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:16.120503 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:16.120462    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx" event={"ID":"2e3e96d1-8201-411c-8e71-132718d75452","Type":"ContainerStarted","Data":"3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c"}
Oct 30 19:06:16.120732 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:16.120569    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx" event={"ID":"2e3e96d1-8201-411c-8e71-132718d75452","Type":"ContainerStarted","Data":"c470587cac4973ee59b21837e29cbce2b0ae38b8f32375c177e86734c4fc4194"}
Oct 30 19:06:18.014742 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:18.014658    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Oct 30 19:06:18.065158 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:18.065122    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:18.132927 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:18.132887    4853 generic.go:356] "Generic (PLEG): container finished" podID="2e3e96d1-8201-411c-8e71-132718d75452" containerID="3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c" exitCode=0
Oct 30 19:06:18.133102 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:18.132948    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx" event={"ID":"2e3e96d1-8201-411c-8e71-132718d75452","Type":"ContainerDied","Data":"3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c"}
Oct 30 19:06:19.148515 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:19.148472    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx" event={"ID":"2e3e96d1-8201-411c-8e71-132718d75452","Type":"ContainerStarted","Data":"b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869"}
Oct 30 19:06:19.355102 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:19.355036    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx" podStartSLOduration=3.039921182 podStartE2EDuration="5.355024542s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.31510336s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:19.308657376 +0000 UTC m=+1685.479014225" watchObservedRunningTime="2026-10-30 19:06:19.355024542 +0000 UTC m=+1685.525381390"
Oct 30 19:06:19.781051 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:19.781020    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:19.876865 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:06:19.876831    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:06:19.945697 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:19.945653    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:20.147368 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:20.147295    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:22.226170 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:22.226137    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:24.781334 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:24.781293    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:24.799379 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:24.799351    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:25.191303 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:25.191273    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:28.014400 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:28.014362    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Oct 30 19:06:33.908152 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:33.908119    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:38.014958 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:38.014919    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Oct 30 19:06:48.014592 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:48.014548    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Oct 30 19:06:49.872819 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:06:49.872776    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:06:54.961659 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:54.961620    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"]
Oct 30 19:06:54.962246 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:54.961890    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" podUID="724ac112-9245-4ae5-aa46-ddc1ea352431" containerName="istio-proxy" containerID="cri-o://a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812" gracePeriod=30
Oct 30 19:06:55.008855 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.008820    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"]
Oct 30 19:06:55.009238 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.009192    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy" containerID="cri-o://8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832" gracePeriod=30
Oct 30 19:06:55.010582 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.010562    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"]
Oct 30 19:06:55.010740 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.010722    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" podUID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" containerName="istio-proxy" containerID="cri-o://f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1" gracePeriod=30
Oct 30 19:06:55.055590 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.055560    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"]
Oct 30 19:06:55.055968 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.055784    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerName="istio-proxy" containerID="cri-o://d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62" gracePeriod=30
Oct 30 19:06:55.302254 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.302167    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:06:55.302254 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.302222    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:55.530354 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.530314    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:06:55.530518 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.530382    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" podUID="724ac112-9245-4ae5-aa46-ddc1ea352431" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:55.781075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.781030    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:06:55.781240 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:55.781096    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" podUID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:58.014311 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.014274    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Oct 30 19:06:58.154822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.154783    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"]
Oct 30 19:06:58.167963 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:06:58.167912    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869" cmd=["/bin/sleep","15"]
Oct 30 19:06:58.168146 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.167980    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:06:58.168146 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:06:58.167997    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx" podUID="2e3e96d1-8201-411c-8e71-132718d75452" containerName="main" containerID="cri-o://b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869"
Oct 30 19:06:58.168146 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.168047    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx" podUID="2e3e96d1-8201-411c-8e71-132718d75452" containerName="main" containerID="cri-o://b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869" gracePeriod=60
Oct 30 19:06:58.180900 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.180863    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx" podUID="2e3e96d1-8201-411c-8e71-132718d75452" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Oct 30 19:06:58.196742 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.196714    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:58.421007 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.420972    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:58.504071 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.504050    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:58.685491 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.685462    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3e96d1-8201-411c-8e71-132718d75452-kube-api-access-nmwks\" (UniqueName: \"kubernetes.io/projected/2e3e96d1-8201-411c-8e71-132718d75452-kube-api-access-nmwks\") pod \"2e3e96d1-8201-411c-8e71-132718d75452\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") "
Oct 30 19:06:58.685709 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.685500    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-dshm\") pod \"2e3e96d1-8201-411c-8e71-132718d75452\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") "
Oct 30 19:06:58.685709 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.685584    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-tmp-dir\") pod \"2e3e96d1-8201-411c-8e71-132718d75452\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") "
Oct 30 19:06:58.685709 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.685660    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-model-cache\") pod \"2e3e96d1-8201-411c-8e71-132718d75452\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") "
Oct 30 19:06:58.685709 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.685699    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3e96d1-8201-411c-8e71-132718d75452-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e3e96d1-8201-411c-8e71-132718d75452-cabundle-cert\") pod \"2e3e96d1-8201-411c-8e71-132718d75452\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") "
Oct 30 19:06:58.685949 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.685741    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-kserve-provision-location\") pod \"2e3e96d1-8201-411c-8e71-132718d75452\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") "
Oct 30 19:06:58.685949 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.685777    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e3e96d1-8201-411c-8e71-132718d75452-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3e96d1-8201-411c-8e71-132718d75452-tls-certs\") pod \"2e3e96d1-8201-411c-8e71-132718d75452\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") "
Oct 30 19:06:58.685949 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.685806    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-home\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-home\") pod \"2e3e96d1-8201-411c-8e71-132718d75452\" (UID: \"2e3e96d1-8201-411c-8e71-132718d75452\") "
Oct 30 19:06:58.685949 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.685922    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-model-cache" pod "2e3e96d1-8201-411c-8e71-132718d75452" (UID: "2e3e96d1-8201-411c-8e71-132718d75452"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.686157 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.685933    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-tmp-dir" pod "2e3e96d1-8201-411c-8e71-132718d75452" (UID: "2e3e96d1-8201-411c-8e71-132718d75452"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.686157 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.686088    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.686157 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.686106    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.686375 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.686308    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3e96d1-8201-411c-8e71-132718d75452-cabundle-cert" pod "2e3e96d1-8201-411c-8e71-132718d75452" (UID: "2e3e96d1-8201-411c-8e71-132718d75452"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:58.686546 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.686493    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-home" pod "2e3e96d1-8201-411c-8e71-132718d75452" (UID: "2e3e96d1-8201-411c-8e71-132718d75452"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.688090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.688063    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3e96d1-8201-411c-8e71-132718d75452-kube-api-access-nmwks" pod "2e3e96d1-8201-411c-8e71-132718d75452" (UID: "2e3e96d1-8201-411c-8e71-132718d75452"). InnerVolumeSpecName "kube-api-access-nmwks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:58.688187 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.688164    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-dshm" pod "2e3e96d1-8201-411c-8e71-132718d75452" (UID: "2e3e96d1-8201-411c-8e71-132718d75452"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.688414 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.688393    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e3e96d1-8201-411c-8e71-132718d75452-tls-certs" pod "2e3e96d1-8201-411c-8e71-132718d75452" (UID: "2e3e96d1-8201-411c-8e71-132718d75452"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:58.718543 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.718488    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-kserve-provision-location" pod "2e3e96d1-8201-411c-8e71-132718d75452" (UID: "2e3e96d1-8201-411c-8e71-132718d75452"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.786933 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.786901    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e3e96d1-8201-411c-8e71-132718d75452-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.786933 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.786929    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.787119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.786946    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3e96d1-8201-411c-8e71-132718d75452-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.787119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.786959    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.787119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.786967    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmwks\" (UniqueName: \"kubernetes.io/projected/2e3e96d1-8201-411c-8e71-132718d75452-kube-api-access-nmwks\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.787119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:58.786975    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e3e96d1-8201-411c-8e71-132718d75452-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:59.352189 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.352155    4853 generic.go:356] "Generic (PLEG): container finished" podID="2e3e96d1-8201-411c-8e71-132718d75452" containerID="b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869" exitCode=0
Oct 30 19:06:59.352189 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.352193    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx" event={"ID":"2e3e96d1-8201-411c-8e71-132718d75452","Type":"ContainerDied","Data":"b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869"}
Oct 30 19:06:59.352758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.352228    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"
Oct 30 19:06:59.352758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.352246    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx" event={"ID":"2e3e96d1-8201-411c-8e71-132718d75452","Type":"ContainerDied","Data":"c470587cac4973ee59b21837e29cbce2b0ae38b8f32375c177e86734c4fc4194"}
Oct 30 19:06:59.352758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.352276    4853 scope.go:122] "RemoveContainer" containerID="b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869"
Oct 30 19:06:59.361383 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.361359    4853 scope.go:122] "RemoveContainer" containerID="3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c"
Oct 30 19:06:59.369665 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.369644    4853 scope.go:122] "RemoveContainer" containerID="b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869"
Oct 30 19:06:59.369928 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:06:59.369907    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869\": container with ID starting with b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869 not found: ID does not exist" containerID="b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869"
Oct 30 19:06:59.370010 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.369943    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869"} err="failed to get container status \"b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869\": rpc error: code = NotFound desc = could not find container \"b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869\": container with ID starting with b35efd61e3cc5e6a44d5a0ebe2e9ef770514e94d1fe8fe42971ebc95962c0869 not found: ID does not exist"
Oct 30 19:06:59.370010 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.369968    4853 scope.go:122] "RemoveContainer" containerID="3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c"
Oct 30 19:06:59.370209 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:06:59.370189    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c\": container with ID starting with 3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c not found: ID does not exist" containerID="3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c"
Oct 30 19:06:59.370244 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.370218    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c"} err="failed to get container status \"3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c\": rpc error: code = NotFound desc = could not find container \"3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c\": container with ID starting with 3fbd099e87a5257def48d905ff1d4e184788e172695d2446959e823e39db590c not found: ID does not exist"
Oct 30 19:06:59.514709 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.514679    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"]
Oct 30 19:06:59.561147 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.561112    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-4x5hx"]
Oct 30 19:06:59.939329 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:06:59.939295    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3e96d1-8201-411c-8e71-132718d75452" path="/var/lib/kubelet/pods/2e3e96d1-8201-411c-8e71-132718d75452/volumes"
Oct 30 19:07:00.296168 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.296144    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 19:07:00.321948 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.321924    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 19:07:00.335892 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.335871    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 19:07:00.338414 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.338394    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 19:07:00.358225 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.358204    4853 generic.go:356] "Generic (PLEG): container finished" podID="724ac112-9245-4ae5-aa46-ddc1ea352431" containerID="a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812" exitCode=0
Oct 30 19:07:00.358588 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.358249    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"
Oct 30 19:07:00.358588 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.358288    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" event={"ID":"724ac112-9245-4ae5-aa46-ddc1ea352431","Type":"ContainerDied","Data":"a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812"}
Oct 30 19:07:00.358588 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.358352    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4" event={"ID":"724ac112-9245-4ae5-aa46-ddc1ea352431","Type":"ContainerDied","Data":"ab1f51e37344c8b6f4b35514c721084069fb45d02fe4b907326eebaeea4b498e"}
Oct 30 19:07:00.358588 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.358370    4853 scope.go:122] "RemoveContainer" containerID="a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812"
Oct 30 19:07:00.361223 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.361199    4853 generic.go:356] "Generic (PLEG): container finished" podID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerID="8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832" exitCode=0
Oct 30 19:07:00.361337 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.361280    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" event={"ID":"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9","Type":"ContainerDied","Data":"8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832"}
Oct 30 19:07:00.361406 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.361340    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw" event={"ID":"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9","Type":"ContainerDied","Data":"0c371f82c20c2ce5ee742cdf4a9ac0f62ac802668150677faaa11b233d2749d8"}
Oct 30 19:07:00.361406 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.361302    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"
Oct 30 19:07:00.362595 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.362577    4853 generic.go:356] "Generic (PLEG): container finished" podID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" containerID="f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1" exitCode=0
Oct 30 19:07:00.362702 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.362623    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"
Oct 30 19:07:00.362702 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.362650    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" event={"ID":"407e5a69-464d-4c90-8ab0-2416b9bfe8c8","Type":"ContainerDied","Data":"f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1"}
Oct 30 19:07:00.362702 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.362685    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk" event={"ID":"407e5a69-464d-4c90-8ab0-2416b9bfe8c8","Type":"ContainerDied","Data":"3fcbc32ddd70cc993cf205690a74df6bd6ac5218464cfe5ff2ef22d570f87bd1"}
Oct 30 19:07:00.364112 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.364092    4853 generic.go:356] "Generic (PLEG): container finished" podID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerID="d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62" exitCode=0
Oct 30 19:07:00.364203 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.364132    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" event={"ID":"079a6eea-fc68-40fd-a4ac-327ac73abe2d","Type":"ContainerDied","Data":"d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62"}
Oct 30 19:07:00.364203 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.364135    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"
Oct 30 19:07:00.364203 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.364166    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6" event={"ID":"079a6eea-fc68-40fd-a4ac-327ac73abe2d","Type":"ContainerDied","Data":"1d221182de67e6cb7dc282a123ee90933c71e1887a99299520e81e76d23c5fea"}
Oct 30 19:07:00.367898 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.367882    4853 scope.go:122] "RemoveContainer" containerID="a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812"
Oct 30 19:07:00.368155 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:07:00.368138    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812\": container with ID starting with a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812 not found: ID does not exist" containerID="a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812"
Oct 30 19:07:00.368199 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.368163    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812"} err="failed to get container status \"a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812\": rpc error: code = NotFound desc = could not find container \"a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812\": container with ID starting with a7f841a76694a65237c97310bcbe89583ca2edc15378935205fee506d0fd9812 not found: ID does not exist"
Oct 30 19:07:00.368199 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.368178    4853 scope.go:122] "RemoveContainer" containerID="8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832"
Oct 30 19:07:00.378659 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.378637    4853 scope.go:122] "RemoveContainer" containerID="dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356"
Oct 30 19:07:00.385932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.385916    4853 scope.go:122] "RemoveContainer" containerID="8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832"
Oct 30 19:07:00.386190 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:07:00.386173    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832\": container with ID starting with 8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832 not found: ID does not exist" containerID="8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832"
Oct 30 19:07:00.386236 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.386196    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832"} err="failed to get container status \"8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832\": rpc error: code = NotFound desc = could not find container \"8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832\": container with ID starting with 8c30a76e0e80bff493a2fafb3645ff081602e02c963139e6ef5fdeec6e401832 not found: ID does not exist"
Oct 30 19:07:00.386236 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.386210    4853 scope.go:122] "RemoveContainer" containerID="dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356"
Oct 30 19:07:00.386462 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:07:00.386445    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356\": container with ID starting with dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356 not found: ID does not exist" containerID="dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356"
Oct 30 19:07:00.386504 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.386471    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356"} err="failed to get container status \"dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356\": rpc error: code = NotFound desc = could not find container \"dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356\": container with ID starting with dab2fada1ae6c3fea1563ac341127c41d0a817ee7e3f5b271e18d9ffe4e61356 not found: ID does not exist"
Oct 30 19:07:00.386504 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.386486    4853 scope.go:122] "RemoveContainer" containerID="f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1"
Oct 30 19:07:00.393748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.393731    4853 scope.go:122] "RemoveContainer" containerID="f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1"
Oct 30 19:07:00.393997 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:07:00.393979    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1\": container with ID starting with f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1 not found: ID does not exist" containerID="f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1"
Oct 30 19:07:00.393997 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.394002    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1"} err="failed to get container status \"f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1\": rpc error: code = NotFound desc = could not find container \"f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1\": container with ID starting with f27574a57c37c2a67005f10f9d53408ac994cd64adab019333b23028e6febfb1 not found: ID does not exist"
Oct 30 19:07:00.394138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.394016    4853 scope.go:122] "RemoveContainer" containerID="d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62"
Oct 30 19:07:00.400173 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400153    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istiod-ca-cert\") pod \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") "
Oct 30 19:07:00.400272 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400198    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-data\") pod \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") "
Oct 30 19:07:00.400272 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400222    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-certs\") pod \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") "
Oct 30 19:07:00.400387 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400330    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-podinfo\") pod \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") "
Oct 30 19:07:00.400492 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400473    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-data" pod "079a6eea-fc68-40fd-a4ac-327ac73abe2d" (UID: "079a6eea-fc68-40fd-a4ac-327ac73abe2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.400561 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400492    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-certs" pod "079a6eea-fc68-40fd-a4ac-327ac73abe2d" (UID: "079a6eea-fc68-40fd-a4ac-327ac73abe2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.400616 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400594    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istiod-ca-cert" pod "079a6eea-fc68-40fd-a4ac-327ac73abe2d" (UID: "079a6eea-fc68-40fd-a4ac-327ac73abe2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:00.400781 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400766    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-credential-socket\") pod \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") "
Oct 30 19:07:00.400836 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400803    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-token\" (UniqueName: \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-token\") pod \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") "
Oct 30 19:07:00.400891 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400850    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-envoy\") pod \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") "
Oct 30 19:07:00.400891 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400888    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-socket\") pod \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") "
Oct 30 19:07:00.401043 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.400912    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-kube-api-access-mv4pz\" (UniqueName: \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-kube-api-access-mv4pz\") pod \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\" (UID: \"079a6eea-fc68-40fd-a4ac-327ac73abe2d\") "
Oct 30 19:07:00.401579 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.401543    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.401668 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.401584    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.401668 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.401602    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.401779 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.401650    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-credential-socket" pod "079a6eea-fc68-40fd-a4ac-327ac73abe2d" (UID: "079a6eea-fc68-40fd-a4ac-327ac73abe2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.402984 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.402953    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-podinfo" pod "079a6eea-fc68-40fd-a4ac-327ac73abe2d" (UID: "079a6eea-fc68-40fd-a4ac-327ac73abe2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:07:00.403785 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.403749    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-kube-api-access-mv4pz" pod "079a6eea-fc68-40fd-a4ac-327ac73abe2d" (UID: "079a6eea-fc68-40fd-a4ac-327ac73abe2d"). InnerVolumeSpecName "kube-api-access-mv4pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:00.405048 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.405022    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-socket" pod "079a6eea-fc68-40fd-a4ac-327ac73abe2d" (UID: "079a6eea-fc68-40fd-a4ac-327ac73abe2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.406677 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.406652    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-envoy" pod "079a6eea-fc68-40fd-a4ac-327ac73abe2d" (UID: "079a6eea-fc68-40fd-a4ac-327ac73abe2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.411127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.410959    4853 scope.go:122] "RemoveContainer" containerID="d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62"
Oct 30 19:07:00.411478 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:07:00.411455    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62\": container with ID starting with d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62 not found: ID does not exist" containerID="d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62"
Oct 30 19:07:00.411567 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.411485    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62"} err="failed to get container status \"d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62\": rpc error: code = NotFound desc = could not find container \"d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62\": container with ID starting with d3bf3605084221c63820d5f88da2c388340d4bb54ea32cb2681aec07b6e9af62 not found: ID does not exist"
Oct 30 19:07:00.412040 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.412020    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-token" pod "079a6eea-fc68-40fd-a4ac-327ac73abe2d" (UID: "079a6eea-fc68-40fd-a4ac-327ac73abe2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:00.502666 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502604    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-data\") pod \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") "
Oct 30 19:07:00.502666 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502641    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istiod-ca-cert\") pod \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") "
Oct 30 19:07:00.502666 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502657    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-data\") pod \"724ac112-9245-4ae5-aa46-ddc1ea352431\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") "
Oct 30 19:07:00.502921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502675    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istiod-ca-cert\") pod \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") "
Oct 30 19:07:00.502921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502697    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-certs\") pod \"724ac112-9245-4ae5-aa46-ddc1ea352431\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") "
Oct 30 19:07:00.502921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502780    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-token\" (UniqueName: \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-token\") pod \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") "
Oct 30 19:07:00.502921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502828    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-kube-api-access-gjjz8\" (UniqueName: \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-kube-api-access-gjjz8\") pod \"724ac112-9245-4ae5-aa46-ddc1ea352431\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") "
Oct 30 19:07:00.502921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502867    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-socket\") pod \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") "
Oct 30 19:07:00.502921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502894    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-token\" (UniqueName: \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-token\") pod \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") "
Oct 30 19:07:00.502921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502921    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-envoy\") pod \"724ac112-9245-4ae5-aa46-ddc1ea352431\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") "
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502937    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-certs" pod "724ac112-9245-4ae5-aa46-ddc1ea352431" (UID: "724ac112-9245-4ae5-aa46-ddc1ea352431"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502943    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-data" pod "407e5a69-464d-4c90-8ab0-2416b9bfe8c8" (UID: "407e5a69-464d-4c90-8ab0-2416b9bfe8c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502953    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-data" pod "724ac112-9245-4ae5-aa46-ddc1ea352431" (UID: "724ac112-9245-4ae5-aa46-ddc1ea352431"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.502973    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-credential-socket\") pod \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") "
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503016    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-kube-api-access-9bjv6\" (UniqueName: \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-kube-api-access-9bjv6\") pod \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") "
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503054    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-podinfo\") pod \"724ac112-9245-4ae5-aa46-ddc1ea352431\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") "
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503072    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istiod-ca-cert" pod "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" (UID: "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503097    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istiod-ca-cert" pod "407e5a69-464d-4c90-8ab0-2416b9bfe8c8" (UID: "407e5a69-464d-4c90-8ab0-2416b9bfe8c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503099    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-credential-socket\") pod \"724ac112-9245-4ae5-aa46-ddc1ea352431\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") "
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503242    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-socket" pod "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" (UID: "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503272    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-token\" (UniqueName: \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-token\") pod \"724ac112-9245-4ae5-aa46-ddc1ea352431\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") "
Oct 30 19:07:00.503328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503313    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-envoy\") pod \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503349    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-envoy\") pod \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503376    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-podinfo\") pod \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503408    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-certs\") pod \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503411    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-credential-socket" pod "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" (UID: "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503435    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-data\") pod \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\" (UID: \"3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503477    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-credential-socket\") pod \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503505    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-socket\") pod \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503550    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-certs\") pod \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503593    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-podinfo\") pod \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503630    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-kube-api-access-v7d6t\" (UniqueName: \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-kube-api-access-v7d6t\") pod \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\" (UID: \"407e5a69-464d-4c90-8ab0-2416b9bfe8c8\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503670    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724ac112-9245-4ae5-aa46-ddc1ea352431-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724ac112-9245-4ae5-aa46-ddc1ea352431-istiod-ca-cert\") pod \"724ac112-9245-4ae5-aa46-ddc1ea352431\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503716    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-socket\") pod \"724ac112-9245-4ae5-aa46-ddc1ea352431\" (UID: \"724ac112-9245-4ae5-aa46-ddc1ea352431\") "
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503854    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-credential-socket" pod "724ac112-9245-4ae5-aa46-ddc1ea352431" (UID: "724ac112-9245-4ae5-aa46-ddc1ea352431"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.503984    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504007    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504023    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504037    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504053    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504067    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504081    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504094    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504108    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504122    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.504127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504136    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.505372 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504150    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079a6eea-fc68-40fd-a4ac-327ac73abe2d-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.505372 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504236    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-socket" pod "724ac112-9245-4ae5-aa46-ddc1ea352431" (UID: "724ac112-9245-4ae5-aa46-ddc1ea352431"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.505372 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504249    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv4pz\" (UniqueName: \"kubernetes.io/projected/079a6eea-fc68-40fd-a4ac-327ac73abe2d-kube-api-access-mv4pz\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.505372 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.504265    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.505645 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.505618    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-credential-socket" pod "407e5a69-464d-4c90-8ab0-2416b9bfe8c8" (UID: "407e5a69-464d-4c90-8ab0-2416b9bfe8c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.506156 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.506126    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-envoy" pod "724ac112-9245-4ae5-aa46-ddc1ea352431" (UID: "724ac112-9245-4ae5-aa46-ddc1ea352431"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.506416 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.506377    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-certs" pod "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" (UID: "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.506487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.506449    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-kube-api-access-9bjv6" pod "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" (UID: "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9"). InnerVolumeSpecName "kube-api-access-9bjv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:00.506708 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.506678    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-token" pod "724ac112-9245-4ae5-aa46-ddc1ea352431" (UID: "724ac112-9245-4ae5-aa46-ddc1ea352431"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:00.506708 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.506693    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-socket" pod "407e5a69-464d-4c90-8ab0-2416b9bfe8c8" (UID: "407e5a69-464d-4c90-8ab0-2416b9bfe8c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.507029 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.507003    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724ac112-9245-4ae5-aa46-ddc1ea352431-istiod-ca-cert" pod "724ac112-9245-4ae5-aa46-ddc1ea352431" (UID: "724ac112-9245-4ae5-aa46-ddc1ea352431"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:00.507295 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.507268    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-certs" pod "407e5a69-464d-4c90-8ab0-2416b9bfe8c8" (UID: "407e5a69-464d-4c90-8ab0-2416b9bfe8c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.507477 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.507454    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-data" pod "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" (UID: "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.507627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.507603    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-kube-api-access-gjjz8" pod "724ac112-9245-4ae5-aa46-ddc1ea352431" (UID: "724ac112-9245-4ae5-aa46-ddc1ea352431"). InnerVolumeSpecName "kube-api-access-gjjz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:00.507705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.507685    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-token" pod "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" (UID: "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:00.508030 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.508004    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-podinfo" pod "724ac112-9245-4ae5-aa46-ddc1ea352431" (UID: "724ac112-9245-4ae5-aa46-ddc1ea352431"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:07:00.508214 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.508185    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-envoy" pod "407e5a69-464d-4c90-8ab0-2416b9bfe8c8" (UID: "407e5a69-464d-4c90-8ab0-2416b9bfe8c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.508296 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.508217    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-podinfo" pod "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" (UID: "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:07:00.508911 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.508885    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-podinfo" pod "407e5a69-464d-4c90-8ab0-2416b9bfe8c8" (UID: "407e5a69-464d-4c90-8ab0-2416b9bfe8c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:07:00.508911 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.508893    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-envoy" pod "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" (UID: "3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:00.509070 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.509055    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-kube-api-access-v7d6t" pod "407e5a69-464d-4c90-8ab0-2416b9bfe8c8" (UID: "407e5a69-464d-4c90-8ab0-2416b9bfe8c8"). InnerVolumeSpecName "kube-api-access-v7d6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:00.509753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.509735    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-token" pod "407e5a69-464d-4c90-8ab0-2416b9bfe8c8" (UID: "407e5a69-464d-4c90-8ab0-2416b9bfe8c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:00.605133 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605099    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605133 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605134    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605148    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605161    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605175    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605189    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605204    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605219    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605234    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7d6t\" (UniqueName: \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-kube-api-access-v7d6t\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605249    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/724ac112-9245-4ae5-aa46-ddc1ea352431-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605262    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605275    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605288    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjjz8\" (UniqueName: \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-kube-api-access-gjjz8\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605301    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605314    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605702 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605327    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bjv6\" (UniqueName: \"kubernetes.io/projected/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9-kube-api-access-9bjv6\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605702 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605341    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605702 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605353    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724ac112-9245-4ae5-aa46-ddc1ea352431-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.605702 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.605366    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/407e5a69-464d-4c90-8ab0-2416b9bfe8c8-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:00.962474 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.962436    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"]
Oct 30 19:07:00.971843 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:00.971814    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-6c8rk"]
Oct 30 19:07:01.078182 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:01.078154    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"]
Oct 30 19:07:01.087241 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:01.087217    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fdnbw"]
Oct 30 19:07:01.194434 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:01.194396    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"]
Oct 30 19:07:01.200271 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:01.200245    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2lqh4"]
Oct 30 19:07:01.306240 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:01.306167    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"]
Oct 30 19:07:01.311719 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:01.311686    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bjdf6"]
Oct 30 19:07:01.940154 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:01.940118    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" path="/var/lib/kubelet/pods/079a6eea-fc68-40fd-a4ac-327ac73abe2d/volumes"
Oct 30 19:07:01.940725 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:01.940706    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" path="/var/lib/kubelet/pods/3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9/volumes"
Oct 30 19:07:01.941270 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:01.941254    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" path="/var/lib/kubelet/pods/407e5a69-464d-4c90-8ab0-2416b9bfe8c8/volumes"
Oct 30 19:07:01.942254 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:01.942236    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724ac112-9245-4ae5-aa46-ddc1ea352431" path="/var/lib/kubelet/pods/724ac112-9245-4ae5-aa46-ddc1ea352431/volumes"
Oct 30 19:07:08.015007 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:08.014956    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Oct 30 19:07:18.024304 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:18.024258    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:07:18.031946 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:18.031915    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:07:19.871607 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:07:19.871575    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:07:25.264719 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:25.264691    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"]
Oct 30 19:07:25.340121 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:25.340093    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:25.551785 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:25.551692    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:40.288631 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:40.288557    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main" containerID="cri-o://e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b" gracePeriod=45
Oct 30 19:07:40.928959 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:40.928936    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:07:41.007084 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007028    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0272eff8-6572-4933-8cbe-a6db1d967ad3-kube-api-access-rz6sd\" (UniqueName: \"kubernetes.io/projected/0272eff8-6572-4933-8cbe-a6db1d967ad3-kube-api-access-rz6sd\") pod \"0272eff8-6572-4933-8cbe-a6db1d967ad3\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") "
Oct 30 19:07:41.007084 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007067    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-dshm\") pod \"0272eff8-6572-4933-8cbe-a6db1d967ad3\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") "
Oct 30 19:07:41.007204 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007088    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-model-cache\") pod \"0272eff8-6572-4933-8cbe-a6db1d967ad3\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") "
Oct 30 19:07:41.007204 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007136    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-kserve-provision-location\") pod \"0272eff8-6572-4933-8cbe-a6db1d967ad3\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") "
Oct 30 19:07:41.007204 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007164    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-home\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-home\") pod \"0272eff8-6572-4933-8cbe-a6db1d967ad3\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") "
Oct 30 19:07:41.007204 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007196    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0272eff8-6572-4933-8cbe-a6db1d967ad3-tls-certs\" (UniqueName: \"kubernetes.io/secret/0272eff8-6572-4933-8cbe-a6db1d967ad3-tls-certs\") pod \"0272eff8-6572-4933-8cbe-a6db1d967ad3\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") "
Oct 30 19:07:41.007404 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007239    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-tmp-dir\") pod \"0272eff8-6572-4933-8cbe-a6db1d967ad3\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") "
Oct 30 19:07:41.007404 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007271    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0272eff8-6572-4933-8cbe-a6db1d967ad3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0272eff8-6572-4933-8cbe-a6db1d967ad3-cabundle-cert\") pod \"0272eff8-6572-4933-8cbe-a6db1d967ad3\" (UID: \"0272eff8-6572-4933-8cbe-a6db1d967ad3\") "
Oct 30 19:07:41.007404 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007384    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-model-cache" pod "0272eff8-6572-4933-8cbe-a6db1d967ad3" (UID: "0272eff8-6572-4933-8cbe-a6db1d967ad3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:41.007587 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007475    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:41.007755 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007732    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0272eff8-6572-4933-8cbe-a6db1d967ad3-cabundle-cert" pod "0272eff8-6572-4933-8cbe-a6db1d967ad3" (UID: "0272eff8-6572-4933-8cbe-a6db1d967ad3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:41.007977 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.007952    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-home" pod "0272eff8-6572-4933-8cbe-a6db1d967ad3" (UID: "0272eff8-6572-4933-8cbe-a6db1d967ad3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:41.010007 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.009981    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-dshm" pod "0272eff8-6572-4933-8cbe-a6db1d967ad3" (UID: "0272eff8-6572-4933-8cbe-a6db1d967ad3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:41.010110 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.010040    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0272eff8-6572-4933-8cbe-a6db1d967ad3-tls-certs" pod "0272eff8-6572-4933-8cbe-a6db1d967ad3" (UID: "0272eff8-6572-4933-8cbe-a6db1d967ad3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:41.010110 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.010065    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0272eff8-6572-4933-8cbe-a6db1d967ad3-kube-api-access-rz6sd" pod "0272eff8-6572-4933-8cbe-a6db1d967ad3" (UID: "0272eff8-6572-4933-8cbe-a6db1d967ad3"). InnerVolumeSpecName "kube-api-access-rz6sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:41.022102 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.022057    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-tmp-dir" pod "0272eff8-6572-4933-8cbe-a6db1d967ad3" (UID: "0272eff8-6572-4933-8cbe-a6db1d967ad3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:41.029084 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.029055    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-kserve-provision-location" pod "0272eff8-6572-4933-8cbe-a6db1d967ad3" (UID: "0272eff8-6572-4933-8cbe-a6db1d967ad3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:41.108322 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.108297    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:41.108322 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.108318    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:41.108438 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.108328    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0272eff8-6572-4933-8cbe-a6db1d967ad3-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:41.108438 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.108337    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:41.108438 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.108344    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0272eff8-6572-4933-8cbe-a6db1d967ad3-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:41.108438 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.108352    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz6sd\" (UniqueName: \"kubernetes.io/projected/0272eff8-6572-4933-8cbe-a6db1d967ad3-kube-api-access-rz6sd\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:41.108438 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.108360    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0272eff8-6572-4933-8cbe-a6db1d967ad3-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:41.559475 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.559441    4853 generic.go:356] "Generic (PLEG): container finished" podID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerID="e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b" exitCode=0
Oct 30 19:07:41.559921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.559484    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" event={"ID":"0272eff8-6572-4933-8cbe-a6db1d967ad3","Type":"ContainerDied","Data":"e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b"}
Oct 30 19:07:41.559921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.559511    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"
Oct 30 19:07:41.559921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.559550    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv" event={"ID":"0272eff8-6572-4933-8cbe-a6db1d967ad3","Type":"ContainerDied","Data":"a5c082ec318b094503e9edc73b278e6786b3a8894edd3e9a606221303e724b9d"}
Oct 30 19:07:41.559921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.559581    4853 scope.go:122] "RemoveContainer" containerID="e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b"
Oct 30 19:07:41.570049 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.570030    4853 scope.go:122] "RemoveContainer" containerID="6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11"
Oct 30 19:07:41.578001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.577981    4853 scope.go:122] "RemoveContainer" containerID="e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b"
Oct 30 19:07:41.578278 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:07:41.578257    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b\": container with ID starting with e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b not found: ID does not exist" containerID="e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b"
Oct 30 19:07:41.578323 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.578287    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b"} err="failed to get container status \"e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b\": rpc error: code = NotFound desc = could not find container \"e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b\": container with ID starting with e9c96d0225243d1d00be5c349ee3959ce1b1a1ef520f48cfb095e6a6bd04c51b not found: ID does not exist"
Oct 30 19:07:41.578323 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.578308    4853 scope.go:122] "RemoveContainer" containerID="6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11"
Oct 30 19:07:41.578598 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:07:41.578575    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11\": container with ID starting with 6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11 not found: ID does not exist" containerID="6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11"
Oct 30 19:07:41.578672 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.578603    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11"} err="failed to get container status \"6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11\": rpc error: code = NotFound desc = could not find container \"6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11\": container with ID starting with 6b01920f6aaf15c0409b020e90b003e19c19d58a753d87b441b4b88960cd4e11 not found: ID does not exist"
Oct 30 19:07:41.718970 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.718935    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"]
Oct 30 19:07:41.765348 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.765322    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-54976cfdb7-5lsrv"]
Oct 30 19:07:41.938674 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:41.938647    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" path="/var/lib/kubelet/pods/0272eff8-6572-4933-8cbe-a6db1d967ad3/volumes"
Oct 30 19:07:45.713658 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:45.713610    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:45.942760 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:45.942707    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:49.873708 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:07:49.873669    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:07:52.236652 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236624    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"]
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236886    4853 cpu_manager.go:414] "containerMap: removing container" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerName="istio-proxy"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236897    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerName="istio-proxy"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236906    4853 cpu_manager.go:414] "containerMap: removing container" podUID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" containerName="istio-proxy"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236911    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" containerName="istio-proxy"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236919    4853 cpu_manager.go:414] "containerMap: removing container" podUID="2e3e96d1-8201-411c-8e71-132718d75452" containerName="main"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236925    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3e96d1-8201-411c-8e71-132718d75452" containerName="main"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236932    4853 cpu_manager.go:414] "containerMap: removing container" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236936    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236944    4853 cpu_manager.go:414] "containerMap: removing container" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236949    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236955    4853 cpu_manager.go:414] "containerMap: removing container" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236960    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236977    4853 cpu_manager.go:414] "containerMap: removing container" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="storage-initializer"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236981    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="storage-initializer"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236987    4853 cpu_manager.go:414] "containerMap: removing container" podUID="724ac112-9245-4ae5-aa46-ddc1ea352431" containerName="istio-proxy"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.236994    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724ac112-9245-4ae5-aa46-ddc1ea352431" containerName="istio-proxy"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.237002    4853 cpu_manager.go:414] "containerMap: removing container" podUID="2e3e96d1-8201-411c-8e71-132718d75452" containerName="storage-initializer"
Oct 30 19:07:52.237022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.237006    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3e96d1-8201-411c-8e71-132718d75452" containerName="storage-initializer"
Oct 30 19:07:52.237552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.237052    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="724ac112-9245-4ae5-aa46-ddc1ea352431" containerName="istio-proxy"
Oct 30 19:07:52.237552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.237059    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="079a6eea-fc68-40fd-a4ac-327ac73abe2d" containerName="istio-proxy"
Oct 30 19:07:52.237552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.237065    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="407e5a69-464d-4c90-8ab0-2416b9bfe8c8" containerName="istio-proxy"
Oct 30 19:07:52.237552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.237070    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3e96d1-8201-411c-8e71-132718d75452" containerName="main"
Oct 30 19:07:52.237552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.237076    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy"
Oct 30 19:07:52.237552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.237081    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9fa396-d9f2-4d45-a73d-f5c9b5a2caa9" containerName="istio-proxy"
Oct 30 19:07:52.237552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.237086    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="0272eff8-6572-4933-8cbe-a6db1d967ad3" containerName="main"
Oct 30 19:07:52.241183 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.241163    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.338035 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.338009    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"]
Oct 30 19:07:52.339254 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.339228    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-4w8ls\""
Oct 30 19:07:52.339408 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.339366    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Oct 30 19:07:52.339408 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.339369    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Oct 30 19:07:52.339408 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.339402    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Oct 30 19:07:52.386829 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.386804    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.386925 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.386833    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.386925 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.386848    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.387011 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.386991    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.387056 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.387029    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khp57\" (UniqueName: \"kubernetes.io/projected/5432bb84-ee7c-4a2a-8658-00bc80c873f3-kube-api-access-khp57\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.387112 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.387054    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.416207 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.416181    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:52.487988 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.487921    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.487988 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.487969    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khp57\" (UniqueName: \"kubernetes.io/projected/5432bb84-ee7c-4a2a-8658-00bc80c873f3-kube-api-access-khp57\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.487988 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.487985    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.488155 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.488015    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.488155 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.488031    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.488155 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.488047    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.488380 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.488358    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.488454 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.488389    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.488515 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.488472    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.490446 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.490421    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.490707 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.490690    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.545094 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.545067    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khp57\" (UniqueName: \"kubernetes.io/projected/5432bb84-ee7c-4a2a-8658-00bc80c873f3-kube-api-access-khp57\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.553893 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.553865    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:52.871249 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.871219    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:52.923994 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:52.923962    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"]
Oct 30 19:07:52.930125 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:07:52.930094    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5432bb84_ee7c_4a2a_8658_00bc80c873f3.slice/crio-50966e6a7025e8b1eed81d1fba7d4fa95324a18b2f32631e4dd5fa41c224aac2 WatchSource:0}: Error finding container 50966e6a7025e8b1eed81d1fba7d4fa95324a18b2f32631e4dd5fa41c224aac2: Status 404 returned error can't find the container with id 50966e6a7025e8b1eed81d1fba7d4fa95324a18b2f32631e4dd5fa41c224aac2
Oct 30 19:07:53.623145 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:53.623108    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz" event={"ID":"5432bb84-ee7c-4a2a-8658-00bc80c873f3","Type":"ContainerStarted","Data":"c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539"}
Oct 30 19:07:53.623615 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:53.623165    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz" event={"ID":"5432bb84-ee7c-4a2a-8658-00bc80c873f3","Type":"ContainerStarted","Data":"50966e6a7025e8b1eed81d1fba7d4fa95324a18b2f32631e4dd5fa41c224aac2"}
Oct 30 19:07:53.731843 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:53.731792    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz" podStartSLOduration=1.73177852 podStartE2EDuration="1.73177852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:53.730547446 +0000 UTC m=+1779.900904284" watchObservedRunningTime="2026-10-30 19:07:53.73177852 +0000 UTC m=+1779.902135345"
Oct 30 19:07:54.318049 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:54.318001    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:54.554091 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:54.554068    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:07:58.584381 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:07:58.584346    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:01.106146 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:01.106081    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:01.320252 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:01.320202    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:02.554220 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:02.554186    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:08:02.567315 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:02.567289    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:08:02.673508 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:02.673484    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:08:13.489297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:13.489234    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:13.703436 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:13.703405    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:18.699762 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.699727    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"]
Oct 30 19:08:18.703837 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.703810    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.754926 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.754897    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"]
Oct 30 19:08:18.755174 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.755158    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Oct 30 19:08:18.800195 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.800168    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Oct 30 19:08:18.800195 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.800175    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:08:18.800367 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.800237    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Oct 30 19:08:18.800439 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.800424    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Oct 30 19:08:18.800544 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.800515    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-5rzb2\""
Oct 30 19:08:18.807775 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.807751    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:18.868845 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.868821    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tmp-dir\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.869180 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.868853    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-home\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.869180 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.868874    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snsxk\" (UniqueName: \"kubernetes.io/projected/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kube-api-access-snsxk\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.869180 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.869016    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91a08878-a1d1-4eee-bb6b-4a96c329e38d-cabundle-cert\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.869180 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.869104    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tls-certs\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.869180 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.869154    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-dshm\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.869180 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.869230    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-model-cache\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.869766 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.869273    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kserve-provision-location\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970096 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970031    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kserve-provision-location\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970096 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970063    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tmp-dir\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970096 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970084    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-home\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970298 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970104    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snsxk\" (UniqueName: \"kubernetes.io/projected/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kube-api-access-snsxk\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970298 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970130    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91a08878-a1d1-4eee-bb6b-4a96c329e38d-cabundle-cert\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970298 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970161    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tls-certs\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970298 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970198    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-dshm\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970298 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970221    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-model-cache\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970524 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970460    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kserve-provision-location\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970524 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970501    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-home\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970665 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970617    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tmp-dir\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.970665 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970649    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-model-cache\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.971000 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.970963    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91a08878-a1d1-4eee-bb6b-4a96c329e38d-cabundle-cert\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.972632 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.972613    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-dshm\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:18.973003 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:18.972984    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tls-certs\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:19.025356 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:19.025336    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snsxk\" (UniqueName: \"kubernetes.io/projected/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kube-api-access-snsxk\") pod \"auth-disabled-test-kserve-86df64bdcd-pcrzc\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:19.319489 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:19.319423    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:19.490968 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:19.490928    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:19.800366 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:19.800336    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"]
Oct 30 19:08:19.858734 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:08:19.858699    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a08878_a1d1_4eee_bb6b_4a96c329e38d.slice/crio-cb4c7dd7eb51713440962a1093a62fd0552149fd50575ace60eb31e0f1a93b7b WatchSource:0}: Error finding container cb4c7dd7eb51713440962a1093a62fd0552149fd50575ace60eb31e0f1a93b7b: Status 404 returned error can't find the container with id cb4c7dd7eb51713440962a1093a62fd0552149fd50575ace60eb31e0f1a93b7b
Oct 30 19:08:19.881091 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:08:19.881057    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:08:20.749466 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:20.749381    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" event={"ID":"91a08878-a1d1-4eee-bb6b-4a96c329e38d","Type":"ContainerStarted","Data":"bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec"}
Oct 30 19:08:20.749466 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:20.749432    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" event={"ID":"91a08878-a1d1-4eee-bb6b-4a96c329e38d","Type":"ContainerStarted","Data":"cb4c7dd7eb51713440962a1093a62fd0552149fd50575ace60eb31e0f1a93b7b"}
Oct 30 19:08:21.753778 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:21.753728    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:22.761302 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:22.761269    4853 generic.go:356] "Generic (PLEG): container finished" podID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerID="bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec" exitCode=0
Oct 30 19:08:22.761779 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:22.761302    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" event={"ID":"91a08878-a1d1-4eee-bb6b-4a96c329e38d","Type":"ContainerDied","Data":"bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec"}
Oct 30 19:08:23.768885 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:23.768845    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" event={"ID":"91a08878-a1d1-4eee-bb6b-4a96c329e38d","Type":"ContainerStarted","Data":"f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af"}
Oct 30 19:08:23.976208 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:23.976146    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podStartSLOduration=3.819191414 podStartE2EDuration="5.976133494s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.15694208s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:23.975450122 +0000 UTC m=+1810.145806980" watchObservedRunningTime="2026-10-30 19:08:23.976133494 +0000 UTC m=+1810.146490341"
Oct 30 19:08:24.320183 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:24.320137    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:26.197203 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:26.197164    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:28.202287 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:28.202252    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:28.414034 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:28.413996    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:29.319955 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:29.319918    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:08:29.321155 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:29.321130    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 19:08:31.890190 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:31.890156    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"]
Oct 30 19:08:31.901238 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:08:31.901187    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539" cmd=["/bin/sleep","15"]
Oct 30 19:08:31.901378 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:31.901266    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:08:31.901378 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:08:31.901288    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz" podUID="5432bb84-ee7c-4a2a-8658-00bc80c873f3" containerName="main" containerID="cri-o://c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539"
Oct 30 19:08:31.901378 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:31.901338    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz" podUID="5432bb84-ee7c-4a2a-8658-00bc80c873f3" containerName="main" containerID="cri-o://c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539" gracePeriod=60
Oct 30 19:08:31.963991 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:31.963961    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:32.151718 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.151660    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:08:32.175791 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.175756    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:32.284192 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.284164    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5432bb84-ee7c-4a2a-8658-00bc80c873f3-kube-api-access-khp57\" (UniqueName: \"kubernetes.io/projected/5432bb84-ee7c-4a2a-8658-00bc80c873f3-kube-api-access-khp57\") pod \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") "
Oct 30 19:08:32.284352 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.284209    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-dshm\") pod \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") "
Oct 30 19:08:32.284352 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.284243    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-model-cache\") pod \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") "
Oct 30 19:08:32.284352 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.284267    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tmp-dir\") pod \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") "
Oct 30 19:08:32.284352 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.284283    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-home\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-home\") pod \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") "
Oct 30 19:08:32.284352 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.284336    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tls-certs\") pod \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\" (UID: \"5432bb84-ee7c-4a2a-8658-00bc80c873f3\") "
Oct 30 19:08:32.284658 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.284595    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-model-cache" pod "5432bb84-ee7c-4a2a-8658-00bc80c873f3" (UID: "5432bb84-ee7c-4a2a-8658-00bc80c873f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:32.284717 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.284658    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tmp-dir" pod "5432bb84-ee7c-4a2a-8658-00bc80c873f3" (UID: "5432bb84-ee7c-4a2a-8658-00bc80c873f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:32.284900 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.284874    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-home" pod "5432bb84-ee7c-4a2a-8658-00bc80c873f3" (UID: "5432bb84-ee7c-4a2a-8658-00bc80c873f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:32.286674 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.286652    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-dshm" pod "5432bb84-ee7c-4a2a-8658-00bc80c873f3" (UID: "5432bb84-ee7c-4a2a-8658-00bc80c873f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:32.287148 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.287124    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5432bb84-ee7c-4a2a-8658-00bc80c873f3-kube-api-access-khp57" pod "5432bb84-ee7c-4a2a-8658-00bc80c873f3" (UID: "5432bb84-ee7c-4a2a-8658-00bc80c873f3"). InnerVolumeSpecName "kube-api-access-khp57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:32.287240 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.287195    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tls-certs" pod "5432bb84-ee7c-4a2a-8658-00bc80c873f3" (UID: "5432bb84-ee7c-4a2a-8658-00bc80c873f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:08:32.385654 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.385627    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:32.385654 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.385649    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:32.385807 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.385679    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:32.385807 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.385689    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5432bb84-ee7c-4a2a-8658-00bc80c873f3-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:32.385807 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.385697    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5432bb84-ee7c-4a2a-8658-00bc80c873f3-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:32.385807 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.385706    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khp57\" (UniqueName: \"kubernetes.io/projected/5432bb84-ee7c-4a2a-8658-00bc80c873f3-kube-api-access-khp57\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:32.809711 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.809675    4853 generic.go:356] "Generic (PLEG): container finished" podID="5432bb84-ee7c-4a2a-8658-00bc80c873f3" containerID="c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539" exitCode=0
Oct 30 19:08:32.809913 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.809727    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"
Oct 30 19:08:32.809913 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.809759    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz" event={"ID":"5432bb84-ee7c-4a2a-8658-00bc80c873f3","Type":"ContainerDied","Data":"c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539"}
Oct 30 19:08:32.809913 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.809815    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz" event={"ID":"5432bb84-ee7c-4a2a-8658-00bc80c873f3","Type":"ContainerDied","Data":"50966e6a7025e8b1eed81d1fba7d4fa95324a18b2f32631e4dd5fa41c224aac2"}
Oct 30 19:08:32.809913 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.809839    4853 scope.go:122] "RemoveContainer" containerID="c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539"
Oct 30 19:08:32.819649 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.819579    4853 scope.go:122] "RemoveContainer" containerID="c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539"
Oct 30 19:08:32.819889 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:08:32.819871    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539\": container with ID starting with c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539 not found: ID does not exist" containerID="c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539"
Oct 30 19:08:32.819931 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.819898    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539"} err="failed to get container status \"c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539\": rpc error: code = NotFound desc = could not find container \"c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539\": container with ID starting with c4e79670d699cb8b5d2cffdc3c592f285dd23f4bde86bdfec98e4e2563444539 not found: ID does not exist"
Oct 30 19:08:32.967906 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:32.967873    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"]
Oct 30 19:08:33.014475 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:33.014445    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-82nnz"]
Oct 30 19:08:33.938504 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:33.938474    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5432bb84-ee7c-4a2a-8658-00bc80c873f3" path="/var/lib/kubelet/pods/5432bb84-ee7c-4a2a-8658-00bc80c873f3/volumes"
Oct 30 19:08:39.319849 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:39.319806    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 19:08:49.319891 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:49.319850    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 19:08:49.871901 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:08:49.871851    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:08:58.972607 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:58.972570    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:59.320956 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:08:59.320869    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 19:09:09.320677 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:09.320633    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 19:09:19.320430 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:19.320390    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 19:09:19.437895 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:19.437861    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:19.654539 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:19.654488    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:19.874972 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:09:19.874935    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:09:26.793851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:26.793816    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:27.230502 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:27.230467    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:27.622787 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:27.622753    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:29.320341 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:29.320303    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 19:09:31.386366 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:31.386315    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:33.350636 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:33.350595    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:33.556701 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:33.556660    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:39.320826 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:39.320784    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 19:09:49.320489 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:49.320448    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 19:09:49.872321 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:09:49.872277    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:09:59.262619 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:59.262583    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:59.320336 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:59.320289    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 19:09:59.468462 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:09:59.468427    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:09.316789 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.316755    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"]
Oct 30 19:10:09.317163 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.317077    4853 cpu_manager.go:414] "containerMap: removing container" podUID="5432bb84-ee7c-4a2a-8658-00bc80c873f3" containerName="main"
Oct 30 19:10:09.317163 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.317088    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5432bb84-ee7c-4a2a-8658-00bc80c873f3" containerName="main"
Oct 30 19:10:09.317163 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.317134    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="5432bb84-ee7c-4a2a-8658-00bc80c873f3" containerName="main"
Oct 30 19:10:09.319298 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.319278    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.330782 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.330759    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:10:09.339214 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.339190    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:10:09.415472 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.415447    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Oct 30 19:10:09.415472 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.415459    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-rp9gr\""
Oct 30 19:10:09.415663 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.415499    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Oct 30 19:10:09.415663 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.415565    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Oct 30 19:10:09.417801 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.417784    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"]
Oct 30 19:10:09.461883 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.461857    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:09.508797 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.508773    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.508903 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.508819    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.508969 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.508945    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84407fde-db87-4157-8bf1-ae9389a0e8a7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.509038 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.509009    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjdl\" (UniqueName: \"kubernetes.io/projected/84407fde-db87-4157-8bf1-ae9389a0e8a7-kube-api-access-qcjdl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.509091 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.509068    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.509144 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.509108    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.609970 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.609944    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.610130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.609981    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.610130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.610037    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.610130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.610086    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.610130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.610118    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84407fde-db87-4157-8bf1-ae9389a0e8a7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.610343 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.610143    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjdl\" (UniqueName: \"kubernetes.io/projected/84407fde-db87-4157-8bf1-ae9389a0e8a7-kube-api-access-qcjdl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.610437 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.610416    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.610513 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.610485    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.610704 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.610683    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.612575 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.612554    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.612951 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.612926    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84407fde-db87-4157-8bf1-ae9389a0e8a7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.665770 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.665625    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjdl\" (UniqueName: \"kubernetes.io/projected/84407fde-db87-4157-8bf1-ae9389a0e8a7-kube-api-access-qcjdl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.932048 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.931976    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:09.989345 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:09.989310    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:10.313729 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:10.313640    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"]
Oct 30 19:10:10.322584 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:10:10.322553    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84407fde_db87_4157_8bf1_ae9389a0e8a7.slice/crio-2340573d64df8014b1681d309a1631a6405e980b900a09394096e37f274d5175 WatchSource:0}: Error finding container 2340573d64df8014b1681d309a1631a6405e980b900a09394096e37f274d5175: Status 404 returned error can't find the container with id 2340573d64df8014b1681d309a1631a6405e980b900a09394096e37f274d5175
Oct 30 19:10:11.293479 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:11.293445    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h" event={"ID":"84407fde-db87-4157-8bf1-ae9389a0e8a7","Type":"ContainerStarted","Data":"4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b"}
Oct 30 19:10:11.293694 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:11.293497    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h" event={"ID":"84407fde-db87-4157-8bf1-ae9389a0e8a7","Type":"ContainerStarted","Data":"2340573d64df8014b1681d309a1631a6405e980b900a09394096e37f274d5175"}
Oct 30 19:10:11.402068 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:11.402021    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h" podStartSLOduration=2.402008497 podStartE2EDuration="2.402008497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:11.401614462 +0000 UTC m=+1917.571971310" watchObservedRunningTime="2026-10-30 19:10:11.402008497 +0000 UTC m=+1917.572365345"
Oct 30 19:10:11.932861 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:11.932815    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:16.905588 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:16.905542    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:18.889399 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:18.889369    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"]
Oct 30 19:10:18.976171 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:18.976139    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:19.180830 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:19.180760    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:19.872160 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:10:19.872122    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:10:19.933068 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:19.933043    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:19.946370 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:19.946349    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:20.346186 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:20.346158    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:33.915693 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:33.915649    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" containerID="cri-o://f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af" gracePeriod=45
Oct 30 19:10:34.331903 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.331862    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Oct 30 19:10:34.665389 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.665366    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:10:34.684091 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684055    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tmp-dir\") pod \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") "
Oct 30 19:10:34.684196 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684113    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kube-api-access-snsxk\" (UniqueName: \"kubernetes.io/projected/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kube-api-access-snsxk\") pod \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") "
Oct 30 19:10:34.684196 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684150    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-model-cache\") pod \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") "
Oct 30 19:10:34.684309 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684194    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-home\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-home\") pod \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") "
Oct 30 19:10:34.684368 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684319    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-dshm\") pod \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") "
Oct 30 19:10:34.684427 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684382    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tls-certs\" (UniqueName: \"kubernetes.io/secret/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tls-certs\") pod \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") "
Oct 30 19:10:34.684479 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684447    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a08878-a1d1-4eee-bb6b-4a96c329e38d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91a08878-a1d1-4eee-bb6b-4a96c329e38d-cabundle-cert\") pod \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") "
Oct 30 19:10:34.684479 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684445    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-model-cache" pod "91a08878-a1d1-4eee-bb6b-4a96c329e38d" (UID: "91a08878-a1d1-4eee-bb6b-4a96c329e38d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.684609 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684485    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kserve-provision-location\") pod \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\" (UID: \"91a08878-a1d1-4eee-bb6b-4a96c329e38d\") "
Oct 30 19:10:34.684759 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684738    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.684853 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684815    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a08878-a1d1-4eee-bb6b-4a96c329e38d-cabundle-cert" pod "91a08878-a1d1-4eee-bb6b-4a96c329e38d" (UID: "91a08878-a1d1-4eee-bb6b-4a96c329e38d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:34.685015 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.684959    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-home" pod "91a08878-a1d1-4eee-bb6b-4a96c329e38d" (UID: "91a08878-a1d1-4eee-bb6b-4a96c329e38d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.686803 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.686778    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tls-certs" pod "91a08878-a1d1-4eee-bb6b-4a96c329e38d" (UID: "91a08878-a1d1-4eee-bb6b-4a96c329e38d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:34.687173 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.687145    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-dshm" pod "91a08878-a1d1-4eee-bb6b-4a96c329e38d" (UID: "91a08878-a1d1-4eee-bb6b-4a96c329e38d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.687249 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.687167    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kube-api-access-snsxk" pod "91a08878-a1d1-4eee-bb6b-4a96c329e38d" (UID: "91a08878-a1d1-4eee-bb6b-4a96c329e38d"). InnerVolumeSpecName "kube-api-access-snsxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:34.697164 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.697141    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tmp-dir" pod "91a08878-a1d1-4eee-bb6b-4a96c329e38d" (UID: "91a08878-a1d1-4eee-bb6b-4a96c329e38d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.708294 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.708274    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kserve-provision-location" pod "91a08878-a1d1-4eee-bb6b-4a96c329e38d" (UID: "91a08878-a1d1-4eee-bb6b-4a96c329e38d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.785698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.785673    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91a08878-a1d1-4eee-bb6b-4a96c329e38d-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.785794 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.785699    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.785794 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.785714    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.785794 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.785727    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snsxk\" (UniqueName: \"kubernetes.io/projected/91a08878-a1d1-4eee-bb6b-4a96c329e38d-kube-api-access-snsxk\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.785794 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.785741    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.785794 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.785753    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91a08878-a1d1-4eee-bb6b-4a96c329e38d-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.785794 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:34.785766    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91a08878-a1d1-4eee-bb6b-4a96c329e38d-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:35.403705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.403667    4853 generic.go:356] "Generic (PLEG): container finished" podID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerID="f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af" exitCode=0
Oct 30 19:10:35.404156 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.403730    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"
Oct 30 19:10:35.404156 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.403747    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" event={"ID":"91a08878-a1d1-4eee-bb6b-4a96c329e38d","Type":"ContainerDied","Data":"f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af"}
Oct 30 19:10:35.404156 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.403811    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc" event={"ID":"91a08878-a1d1-4eee-bb6b-4a96c329e38d","Type":"ContainerDied","Data":"cb4c7dd7eb51713440962a1093a62fd0552149fd50575ace60eb31e0f1a93b7b"}
Oct 30 19:10:35.404156 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.403836    4853 scope.go:122] "RemoveContainer" containerID="f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af"
Oct 30 19:10:35.416507 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.416479    4853 scope.go:122] "RemoveContainer" containerID="bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec"
Oct 30 19:10:35.424906 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.424889    4853 scope.go:122] "RemoveContainer" containerID="f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af"
Oct 30 19:10:35.425242 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:10:35.425221    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af\": container with ID starting with f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af not found: ID does not exist" containerID="f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af"
Oct 30 19:10:35.425301 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.425252    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af"} err="failed to get container status \"f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af\": rpc error: code = NotFound desc = could not find container \"f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af\": container with ID starting with f0e6ff24b8fdb1f6ac42ab973a509f085947e70261a664d951dc95a8f6ea02af not found: ID does not exist"
Oct 30 19:10:35.425301 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.425272    4853 scope.go:122] "RemoveContainer" containerID="bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec"
Oct 30 19:10:35.425518 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:10:35.425501    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec\": container with ID starting with bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec not found: ID does not exist" containerID="bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec"
Oct 30 19:10:35.425596 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.425522    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec"} err="failed to get container status \"bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec\": rpc error: code = NotFound desc = could not find container \"bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec\": container with ID starting with bc67bfc2ab6ac5c9e29612ee8ddc7bbdc3dcfce065e1b4f26a3b19cccc44f3ec not found: ID does not exist"
Oct 30 19:10:35.609338 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.609312    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"]
Oct 30 19:10:35.655670 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.655609    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-86df64bdcd-pcrzc"]
Oct 30 19:10:35.939586 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:35.939510    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" path="/var/lib/kubelet/pods/91a08878-a1d1-4eee-bb6b-4a96c329e38d/volumes"
Oct 30 19:10:48.465957 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.465924    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"]
Oct 30 19:10:48.476246 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:10:48.476208    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b" cmd=["/bin/sleep","15"]
Oct 30 19:10:48.476381 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.476259    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:10:48.476381 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:10:48.476275    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h" podUID="84407fde-db87-4157-8bf1-ae9389a0e8a7" containerName="main" containerID="cri-o://4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b"
Oct 30 19:10:48.476381 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.476322    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h" podUID="84407fde-db87-4157-8bf1-ae9389a0e8a7" containerName="main" containerID="cri-o://4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b" gracePeriod=60
Oct 30 19:10:48.538426 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.538395    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:48.766501 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.766462    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:48.820879 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.820858    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:48.882444 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.882424    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84407fde-db87-4157-8bf1-ae9389a0e8a7-kube-api-access-qcjdl\" (UniqueName: \"kubernetes.io/projected/84407fde-db87-4157-8bf1-ae9389a0e8a7-kube-api-access-qcjdl\") pod \"84407fde-db87-4157-8bf1-ae9389a0e8a7\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") "
Oct 30 19:10:48.882591 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.882457    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-model-cache\") pod \"84407fde-db87-4157-8bf1-ae9389a0e8a7\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") "
Oct 30 19:10:48.882591 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.882478    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-tmp-dir\") pod \"84407fde-db87-4157-8bf1-ae9389a0e8a7\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") "
Oct 30 19:10:48.882591 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.882513    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84407fde-db87-4157-8bf1-ae9389a0e8a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/84407fde-db87-4157-8bf1-ae9389a0e8a7-tls-certs\") pod \"84407fde-db87-4157-8bf1-ae9389a0e8a7\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") "
Oct 30 19:10:48.882725 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.882591    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-home\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-home\") pod \"84407fde-db87-4157-8bf1-ae9389a0e8a7\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") "
Oct 30 19:10:48.882725 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.882617    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-dshm\") pod \"84407fde-db87-4157-8bf1-ae9389a0e8a7\" (UID: \"84407fde-db87-4157-8bf1-ae9389a0e8a7\") "
Oct 30 19:10:48.882825 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.882756    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-model-cache" pod "84407fde-db87-4157-8bf1-ae9389a0e8a7" (UID: "84407fde-db87-4157-8bf1-ae9389a0e8a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:48.882825 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.882770    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-tmp-dir" pod "84407fde-db87-4157-8bf1-ae9389a0e8a7" (UID: "84407fde-db87-4157-8bf1-ae9389a0e8a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:48.882939 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.882855    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-home" pod "84407fde-db87-4157-8bf1-ae9389a0e8a7" (UID: "84407fde-db87-4157-8bf1-ae9389a0e8a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:48.882939 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.882907    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:48.882939 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.882926    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:48.884782 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.884757    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-dshm" pod "84407fde-db87-4157-8bf1-ae9389a0e8a7" (UID: "84407fde-db87-4157-8bf1-ae9389a0e8a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:48.884860 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.884828    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84407fde-db87-4157-8bf1-ae9389a0e8a7-kube-api-access-qcjdl" pod "84407fde-db87-4157-8bf1-ae9389a0e8a7" (UID: "84407fde-db87-4157-8bf1-ae9389a0e8a7"). InnerVolumeSpecName "kube-api-access-qcjdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:48.884930 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.884913    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84407fde-db87-4157-8bf1-ae9389a0e8a7-tls-certs" pod "84407fde-db87-4157-8bf1-ae9389a0e8a7" (UID: "84407fde-db87-4157-8bf1-ae9389a0e8a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:48.983988 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.983963    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84407fde-db87-4157-8bf1-ae9389a0e8a7-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:48.983988 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.983984    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:48.983988 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.983993    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84407fde-db87-4157-8bf1-ae9389a0e8a7-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:48.984161 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:48.984002    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcjdl\" (UniqueName: \"kubernetes.io/projected/84407fde-db87-4157-8bf1-ae9389a0e8a7-kube-api-access-qcjdl\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:49.468261 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:49.468228    4853 generic.go:356] "Generic (PLEG): container finished" podID="84407fde-db87-4157-8bf1-ae9389a0e8a7" containerID="4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b" exitCode=0
Oct 30 19:10:49.468726 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:49.468271    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"
Oct 30 19:10:49.468726 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:49.468276    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h" event={"ID":"84407fde-db87-4157-8bf1-ae9389a0e8a7","Type":"ContainerDied","Data":"4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b"}
Oct 30 19:10:49.468726 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:49.468327    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h" event={"ID":"84407fde-db87-4157-8bf1-ae9389a0e8a7","Type":"ContainerDied","Data":"2340573d64df8014b1681d309a1631a6405e980b900a09394096e37f274d5175"}
Oct 30 19:10:49.468726 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:49.468358    4853 scope.go:122] "RemoveContainer" containerID="4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b"
Oct 30 19:10:49.481377 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:49.481228    4853 scope.go:122] "RemoveContainer" containerID="4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b"
Oct 30 19:10:49.481506 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:10:49.481487    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b\": container with ID starting with 4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b not found: ID does not exist" containerID="4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b"
Oct 30 19:10:49.481563 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:49.481515    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b"} err="failed to get container status \"4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b\": rpc error: code = NotFound desc = could not find container \"4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b\": container with ID starting with 4c71dbd41ced43e2e6f48e3b2565c958a43dc3bbcd171e17ecbf50fffcde8c0b not found: ID does not exist"
Oct 30 19:10:49.629934 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:49.629911    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"]
Oct 30 19:10:49.676159 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:49.676141    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b2h74h"]
Oct 30 19:10:49.872272 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:10:49.872239    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:10:49.938845 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:49.938817    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84407fde-db87-4157-8bf1-ae9389a0e8a7" path="/var/lib/kubelet/pods/84407fde-db87-4157-8bf1-ae9389a0e8a7/volumes"
Oct 30 19:10:59.639075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:10:59.639028    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:01.008563 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:01.008504    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:08.453897 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.453858    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:08.668087 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.668044    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:08.928262 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.928231    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"]
Oct 30 19:11:08.928609 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.928592    4853 cpu_manager.go:414] "containerMap: removing container" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main"
Oct 30 19:11:08.928705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.928610    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main"
Oct 30 19:11:08.928705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.928630    4853 cpu_manager.go:414] "containerMap: removing container" podUID="84407fde-db87-4157-8bf1-ae9389a0e8a7" containerName="main"
Oct 30 19:11:08.928705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.928637    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84407fde-db87-4157-8bf1-ae9389a0e8a7" containerName="main"
Oct 30 19:11:08.928705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.928661    4853 cpu_manager.go:414] "containerMap: removing container" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="storage-initializer"
Oct 30 19:11:08.928705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.928668    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="storage-initializer"
Oct 30 19:11:08.928942 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.928765    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a08878-a1d1-4eee-bb6b-4a96c329e38d" containerName="main"
Oct 30 19:11:08.928942 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.928780    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="84407fde-db87-4157-8bf1-ae9389a0e8a7" containerName="main"
Oct 30 19:11:08.931758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.931727    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:08.982416 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:08.982393    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Oct 30 19:11:09.002142 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.002121    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Oct 30 19:11:09.002262 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.002159    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:11:09.022629 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.022607    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-data\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.022744 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.022640    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-envoy\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.022744 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.022664    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-credential-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.022848 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.022744    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-certs\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.022848 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.022785    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wr65\" (UniqueName: \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-kube-api-access-9wr65\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.022958 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.022865    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-podinfo\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.022958 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.022949    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.023063 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.022979    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-token\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.023063 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.023003    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.027730 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.027711    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-9nk5h\""
Oct 30 19:11:09.029393 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.029370    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"]
Oct 30 19:11:09.061168 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.061136    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:09.124242 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.124213    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-data\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.124242 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.124240    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-envoy\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.124435 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.124272    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-credential-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.124435 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.124421    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-certs\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.124573 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.124464    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wr65\" (UniqueName: \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-kube-api-access-9wr65\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.124573 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.124491    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-podinfo\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.124573 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.124557    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.124745 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.124580    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-token\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.124745 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.124602    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.124857 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.124818    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-data\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.125301 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.125278    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-certs\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.125422 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.125398    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-credential-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.125759 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.125731    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.126117 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.126090    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.127589 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.127563    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-envoy\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.127717 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.127695    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-podinfo\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.180015 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.179955    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-token\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.180239 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.180218    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wr65\" (UniqueName: \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-kube-api-access-9wr65\") pod \"router-gateway-1-openshift-default-746559c7f6-5m2nd\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.244815 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.244790    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:09.259682 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.259655    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:09.671171 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.671139    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"]
Oct 30 19:11:09.678342 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:11:09.678316    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb7aac9_3e03_47b0_a6ff_dde43bfa470a.slice/crio-ceedb333586defc28ee617b91e2dbf034590c42f691d4bc3f93a7bc5d15cf460 WatchSource:0}: Error finding container ceedb333586defc28ee617b91e2dbf034590c42f691d4bc3f93a7bc5d15cf460: Status 404 returned error can't find the container with id ceedb333586defc28ee617b91e2dbf034590c42f691d4bc3f93a7bc5d15cf460
Oct 30 19:11:09.680210 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.680174    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:09.680297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.680239    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:09.680297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:09.680266    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:10.569599 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:10.569565    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" event={"ID":"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a","Type":"ContainerStarted","Data":"9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232"}
Oct 30 19:11:10.569775 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:10.569617    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" event={"ID":"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a","Type":"ContainerStarted","Data":"ceedb333586defc28ee617b91e2dbf034590c42f691d4bc3f93a7bc5d15cf460"}
Oct 30 19:11:10.774741 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:10.774688    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" podStartSLOduration=2.774675427 podStartE2EDuration="2.774675427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:10.726885328 +0000 UTC m=+1976.897242176" watchObservedRunningTime="2026-10-30 19:11:10.774675427 +0000 UTC m=+1976.945032275"
Oct 30 19:11:11.245337 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:11.245301    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:11.250184 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:11.250159    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:11.574092 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:11.574015    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:11.574816 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:11.574801    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:13.261733 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:13.261701    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:15.888118 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:15.888084    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:19.871900 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:11:19.871862    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:11:22.107817 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.107788    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"]
Oct 30 19:11:22.112640 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.112622    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.188737 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.188707    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"]
Oct 30 19:11:22.210861 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.210832    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Oct 30 19:11:22.210985 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.210960    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.211032 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.210988    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.211032 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.210999    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-s565d\""
Oct 30 19:11:22.211032 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.211013    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.211201 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.211176    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.211257 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.211241    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwkg\" (UniqueName: \"kubernetes.io/projected/bef7ab57-5946-4d19-a06e-dcd1500a5e79-kube-api-access-tzwkg\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.211316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.211301    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.305754 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.305714    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:22.311748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.311727    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.311838 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.311758    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.311838 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.311779    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.311838 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.311801    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.311977 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.311851    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.311977 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.311882    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwkg\" (UniqueName: \"kubernetes.io/projected/bef7ab57-5946-4d19-a06e-dcd1500a5e79-kube-api-access-tzwkg\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.312141 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.312122    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.312254 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.312237    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.312320 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.312291    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.314362 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.314345    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.314464 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.314448    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.367091 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.367031    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwkg\" (UniqueName: \"kubernetes.io/projected/bef7ab57-5946-4d19-a06e-dcd1500a5e79-kube-api-access-tzwkg\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.428332 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.428306    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:22.645104 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.645030    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:22.801106 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.801077    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"]
Oct 30 19:11:22.807845 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:11:22.807818    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef7ab57_5946_4d19_a06e_dcd1500a5e79.slice/crio-f6ad691c84ea6c9c0fe9e83fe85239f1fae8c3a432690094435f1ff2e0edb95b WatchSource:0}: Error finding container f6ad691c84ea6c9c0fe9e83fe85239f1fae8c3a432690094435f1ff2e0edb95b: Status 404 returned error can't find the container with id f6ad691c84ea6c9c0fe9e83fe85239f1fae8c3a432690094435f1ff2e0edb95b
Oct 30 19:11:22.862455 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:22.862412    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:23.627546 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:23.627504    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k" event={"ID":"bef7ab57-5946-4d19-a06e-dcd1500a5e79","Type":"ContainerStarted","Data":"a19ea810c3618b4e8f0fc0e130ba499dbdb608e464900f481ad759015749ad88"}
Oct 30 19:11:23.627914 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:23.627569    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k" event={"ID":"bef7ab57-5946-4d19-a06e-dcd1500a5e79","Type":"ContainerStarted","Data":"f6ad691c84ea6c9c0fe9e83fe85239f1fae8c3a432690094435f1ff2e0edb95b"}
Oct 30 19:11:23.737029 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:23.736980    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k" podStartSLOduration=1.736966717 podStartE2EDuration="1.736966717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:23.736331472 +0000 UTC m=+1989.906688317" watchObservedRunningTime="2026-10-30 19:11:23.736966717 +0000 UTC m=+1989.907323565"
Oct 30 19:11:24.428729 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:24.428701    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:32.428949 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:32.428913    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:32.441419 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:32.441387    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:32.708356 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:32.708273    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:11:33.702084 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:33.701997    4853 generic.go:356] "Generic (PLEG): container finished" podID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerID="9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232" exitCode=0
Oct 30 19:11:33.702625 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:33.702076    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" event={"ID":"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a","Type":"ContainerDied","Data":"9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232"}
Oct 30 19:11:33.702705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:33.702630    4853 scope.go:122] "RemoveContainer" containerID="9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232"
Oct 30 19:11:33.703712 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:33.703689    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:33.703781 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:33.703749    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:33.703822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:33.703791    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:11:34.708880 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:34.708845    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" event={"ID":"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a","Type":"ContainerStarted","Data":"07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013"}
Oct 30 19:11:35.249555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:35.247859    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:35.249555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:35.247973    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 19:11:36.249950 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:36.249906    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 19:11:37.254955 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:37.254867    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 19:11:37.258492 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:37.258450    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:37.490066 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:37.490027    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:38.253554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:38.247815    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 19:11:39.257555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:39.254859    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 19:11:40.250162 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:40.250129    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:40.250562 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:40.250521    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:40.251228 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:40.251199    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:11:49.874379 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:11:49.874339    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:11:51.544493 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:51.544451    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:59.985310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:11:59.985275    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:06.699884 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.699849    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"]
Oct 30 19:12:06.708938 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:12:06.708891    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="a19ea810c3618b4e8f0fc0e130ba499dbdb608e464900f481ad759015749ad88" cmd=["/bin/sleep","15"]
Oct 30 19:12:06.709119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.708958    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:12:06.709119 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:12:06.708974    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k" podUID="bef7ab57-5946-4d19-a06e-dcd1500a5e79" containerName="main" containerID="cri-o://a19ea810c3618b4e8f0fc0e130ba499dbdb608e464900f481ad759015749ad88"
Oct 30 19:12:06.709119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.709027    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k" podUID="bef7ab57-5946-4d19-a06e-dcd1500a5e79" containerName="main" containerID="cri-o://a19ea810c3618b4e8f0fc0e130ba499dbdb608e464900f481ad759015749ad88" gracePeriod=60
Oct 30 19:12:06.804297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.804270    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:06.860660 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.860623    4853 generic.go:356] "Generic (PLEG): container finished" podID="bef7ab57-5946-4d19-a06e-dcd1500a5e79" containerID="a19ea810c3618b4e8f0fc0e130ba499dbdb608e464900f481ad759015749ad88" exitCode=0
Oct 30 19:12:06.860801 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.860701    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k" event={"ID":"bef7ab57-5946-4d19-a06e-dcd1500a5e79","Type":"ContainerDied","Data":"a19ea810c3618b4e8f0fc0e130ba499dbdb608e464900f481ad759015749ad88"}
Oct 30 19:12:06.957447 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.957381    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:12:06.995611 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.995590    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tmp-dir\") pod \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") "
Oct 30 19:12:06.995719 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.995626    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-model-cache\") pod \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") "
Oct 30 19:12:06.995719 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.995670    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tls-certs\" (UniqueName: \"kubernetes.io/secret/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tls-certs\") pod \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") "
Oct 30 19:12:06.995719 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.995696    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef7ab57-5946-4d19-a06e-dcd1500a5e79-kube-api-access-tzwkg\" (UniqueName: \"kubernetes.io/projected/bef7ab57-5946-4d19-a06e-dcd1500a5e79-kube-api-access-tzwkg\") pod \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") "
Oct 30 19:12:06.995884 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.995736    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-home\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-home\") pod \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") "
Oct 30 19:12:06.995884 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.995843    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tmp-dir" pod "bef7ab57-5946-4d19-a06e-dcd1500a5e79" (UID: "bef7ab57-5946-4d19-a06e-dcd1500a5e79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:06.995987 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.995885    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-dshm\") pod \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\" (UID: \"bef7ab57-5946-4d19-a06e-dcd1500a5e79\") "
Oct 30 19:12:06.995987 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.995919    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-model-cache" pod "bef7ab57-5946-4d19-a06e-dcd1500a5e79" (UID: "bef7ab57-5946-4d19-a06e-dcd1500a5e79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:06.996094 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.996031    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:06.996094 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.996044    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:06.996094 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.996047    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-home" pod "bef7ab57-5946-4d19-a06e-dcd1500a5e79" (UID: "bef7ab57-5946-4d19-a06e-dcd1500a5e79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:06.998035 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.998005    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tls-certs" pod "bef7ab57-5946-4d19-a06e-dcd1500a5e79" (UID: "bef7ab57-5946-4d19-a06e-dcd1500a5e79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:06.998133 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.998063    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-dshm" pod "bef7ab57-5946-4d19-a06e-dcd1500a5e79" (UID: "bef7ab57-5946-4d19-a06e-dcd1500a5e79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:06.998133 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:06.998065    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef7ab57-5946-4d19-a06e-dcd1500a5e79-kube-api-access-tzwkg" pod "bef7ab57-5946-4d19-a06e-dcd1500a5e79" (UID: "bef7ab57-5946-4d19-a06e-dcd1500a5e79"). InnerVolumeSpecName "kube-api-access-tzwkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:07.096567 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:07.096544    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef7ab57-5946-4d19-a06e-dcd1500a5e79-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:07.096567 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:07.096565    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzwkg\" (UniqueName: \"kubernetes.io/projected/bef7ab57-5946-4d19-a06e-dcd1500a5e79-kube-api-access-tzwkg\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:07.096717 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:07.096575    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:07.096717 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:07.096583    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef7ab57-5946-4d19-a06e-dcd1500a5e79-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:07.867755 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:07.867717    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k" event={"ID":"bef7ab57-5946-4d19-a06e-dcd1500a5e79","Type":"ContainerDied","Data":"f6ad691c84ea6c9c0fe9e83fe85239f1fae8c3a432690094435f1ff2e0edb95b"}
Oct 30 19:12:07.868206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:07.867781    4853 scope.go:122] "RemoveContainer" containerID="a19ea810c3618b4e8f0fc0e130ba499dbdb608e464900f481ad759015749ad88"
Oct 30 19:12:07.868206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:07.867734    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"
Oct 30 19:12:08.024867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:08.024841    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"]
Oct 30 19:12:08.071244 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:08.071212    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb4lt8k"]
Oct 30 19:12:09.938688 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:09.938660    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef7ab57-5946-4d19-a06e-dcd1500a5e79" path="/var/lib/kubelet/pods/bef7ab57-5946-4d19-a06e-dcd1500a5e79/volumes"
Oct 30 19:12:11.092408 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:11.092365    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:11.400476 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:11.400391    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:15.993471 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:15.993442    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"]
Oct 30 19:12:15.993924 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:15.993643    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy" containerID="cri-o://07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013" gracePeriod=30
Oct 30 19:12:16.153901 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:16.153869    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:17.072212 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.072182    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"]
Oct 30 19:12:17.072647 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.072543    4853 cpu_manager.go:414] "containerMap: removing container" podUID="bef7ab57-5946-4d19-a06e-dcd1500a5e79" containerName="main"
Oct 30 19:12:17.072647 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.072559    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef7ab57-5946-4d19-a06e-dcd1500a5e79" containerName="main"
Oct 30 19:12:17.072647 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.072612    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef7ab57-5946-4d19-a06e-dcd1500a5e79" containerName="main"
Oct 30 19:12:17.077130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.077110    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.129096 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.129067    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"]
Oct 30 19:12:17.167817 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.167785    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.167943 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.167820    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a81bd8-b6d4-468d-bb5e-95db7854810f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.167943 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.167839    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.167943 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.167904    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a81bd8-b6d4-468d-bb5e-95db7854810f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.168054 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.168001    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.168087 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.168057    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmxxx\" (UniqueName: \"kubernetes.io/projected/73a81bd8-b6d4-468d-bb5e-95db7854810f-kube-api-access-pmxxx\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.168124 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.168084    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.168124 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.168112    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.174133 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.174112    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Oct 30 19:12:17.174288 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.174153    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Oct 30 19:12:17.174288 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.174153    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:12:17.174425 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.174158    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Oct 30 19:12:17.174425 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.174227    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-z7tcw\""
Oct 30 19:12:17.174425 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.174273    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Oct 30 19:12:17.237495 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.237471    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:17.268665 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.268644    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.268767 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.268675    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmxxx\" (UniqueName: \"kubernetes.io/projected/73a81bd8-b6d4-468d-bb5e-95db7854810f-kube-api-access-pmxxx\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.268767 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.268697    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.268767 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.268734    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.268897 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.268776    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.268897 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.268801    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a81bd8-b6d4-468d-bb5e-95db7854810f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.268897 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.268829    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.269082 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.268925    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a81bd8-b6d4-468d-bb5e-95db7854810f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.269139 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.269076    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.269139 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.269108    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.269243 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.269178    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.269316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.269295    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.269629 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.269609    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a81bd8-b6d4-468d-bb5e-95db7854810f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.271180 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.271157    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.271441 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.271424    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a81bd8-b6d4-468d-bb5e-95db7854810f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.327350 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.327332    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmxxx\" (UniqueName: \"kubernetes.io/projected/73a81bd8-b6d4-468d-bb5e-95db7854810f-kube-api-access-pmxxx\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.392974 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.392946    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:17.743186 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.743093    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:17.868392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:17.868358    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"]
Oct 30 19:12:17.912881 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:12:17.912848    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a81bd8_b6d4_468d_bb5e_95db7854810f.slice/crio-cf77f54c7138be853a9bf2709e26b5cb58404b18497c44d823f82b4ae4b8b3b5 WatchSource:0}: Error finding container cf77f54c7138be853a9bf2709e26b5cb58404b18497c44d823f82b4ae4b8b3b5: Status 404 returned error can't find the container with id cf77f54c7138be853a9bf2709e26b5cb58404b18497c44d823f82b4ae4b8b3b5
Oct 30 19:12:18.923361 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:18.923323    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" event={"ID":"73a81bd8-b6d4-468d-bb5e-95db7854810f","Type":"ContainerStarted","Data":"a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8"}
Oct 30 19:12:18.923807 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:18.923386    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" event={"ID":"73a81bd8-b6d4-468d-bb5e-95db7854810f","Type":"ContainerStarted","Data":"cf77f54c7138be853a9bf2709e26b5cb58404b18497c44d823f82b4ae4b8b3b5"}
Oct 30 19:12:19.879689 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:12:19.879650    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:12:20.102409 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:12:20.102380    4853 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a81bd8_b6d4_468d_bb5e_95db7854810f.slice/crio-a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:12:20.933976 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:20.933943    4853 generic.go:356] "Generic (PLEG): container finished" podID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerID="a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8" exitCode=0
Oct 30 19:12:20.934147 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:20.933988    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" event={"ID":"73a81bd8-b6d4-468d-bb5e-95db7854810f","Type":"ContainerDied","Data":"a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8"}
Oct 30 19:12:21.232222 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.232199    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:12:21.304048 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304022    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istiod-ca-cert\") pod \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") "
Oct 30 19:12:21.304308 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304085    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-socket\") pod \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") "
Oct 30 19:12:21.304308 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304127    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-data\") pod \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") "
Oct 30 19:12:21.304308 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304155    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-credential-socket\") pod \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") "
Oct 30 19:12:21.304308 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304192    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-kube-api-access-9wr65\" (UniqueName: \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-kube-api-access-9wr65\") pod \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") "
Oct 30 19:12:21.304308 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304229    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-token\") pod \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") "
Oct 30 19:12:21.304308 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304277    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-podinfo\") pod \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") "
Oct 30 19:12:21.304308 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304308    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-envoy\") pod \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") "
Oct 30 19:12:21.304702 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304341    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-certs\") pod \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\" (UID: \"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a\") "
Oct 30 19:12:21.304702 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304468    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-socket" pod "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" (UID: "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:21.304702 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304493    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-data" pod "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" (UID: "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:21.304702 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304562    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-credential-socket" pod "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" (UID: "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:21.304908 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304702    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.304908 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304717    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-certs" pod "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" (UID: "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:21.304908 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304723    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.304908 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304765    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.304908 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.304780    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istiod-ca-cert" pod "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" (UID: "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:21.307570 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.307514    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-token" pod "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" (UID: "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:21.307570 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.307553    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-kube-api-access-9wr65" pod "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" (UID: "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a"). InnerVolumeSpecName "kube-api-access-9wr65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:21.307742 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.307596    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-podinfo" pod "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" (UID: "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:12:21.307742 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.307638    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-envoy" pod "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" (UID: "eeb7aac9-3e03-47b0-a6ff-dde43bfa470a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:12:21.405791 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.405757    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.405791 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.405785    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wr65\" (UniqueName: \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-kube-api-access-9wr65\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.405791 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.405797    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.406100 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.405806    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.406100 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.405814    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.406100 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.405821    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.843752 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.843714    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:21.941024 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.940986    4853 generic.go:356] "Generic (PLEG): container finished" podID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerID="07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013" exitCode=0
Oct 30 19:12:21.941195 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.941039    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"
Oct 30 19:12:21.941195 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.941076    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" event={"ID":"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a","Type":"ContainerDied","Data":"07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013"}
Oct 30 19:12:21.941195 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.941136    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd" event={"ID":"eeb7aac9-3e03-47b0-a6ff-dde43bfa470a","Type":"ContainerDied","Data":"ceedb333586defc28ee617b91e2dbf034590c42f691d4bc3f93a7bc5d15cf460"}
Oct 30 19:12:21.941195 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.941167    4853 scope.go:122] "RemoveContainer" containerID="07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013"
Oct 30 19:12:21.943362 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.943340    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" event={"ID":"73a81bd8-b6d4-468d-bb5e-95db7854810f","Type":"ContainerStarted","Data":"4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752"}
Oct 30 19:12:21.954587 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.954564    4853 scope.go:122] "RemoveContainer" containerID="9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232"
Oct 30 19:12:21.965953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.965923    4853 scope.go:122] "RemoveContainer" containerID="07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013"
Oct 30 19:12:21.966402 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:12:21.966377    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013\": container with ID starting with 07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013 not found: ID does not exist" containerID="07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013"
Oct 30 19:12:21.966480 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.966413    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013"} err="failed to get container status \"07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013\": rpc error: code = NotFound desc = could not find container \"07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013\": container with ID starting with 07f6784351a4e7b6608207cd777835858d3c827491fde9e546dda59fb74c6013 not found: ID does not exist"
Oct 30 19:12:21.966480 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.966428    4853 scope.go:122] "RemoveContainer" containerID="9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232"
Oct 30 19:12:21.966907 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:12:21.966885    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232\": container with ID starting with 9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232 not found: ID does not exist" containerID="9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232"
Oct 30 19:12:21.967022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:21.966910    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232"} err="failed to get container status \"9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232\": rpc error: code = NotFound desc = could not find container \"9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232\": container with ID starting with 9aa3bef35b59e6ada2ba9eeda67cadf5ed8eef8f5f4885afc1fe0bca8fc9b232 not found: ID does not exist"
Oct 30 19:12:22.104909 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:22.104836    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"]
Oct 30 19:12:22.151488 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:22.151450    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-5m2nd"]
Oct 30 19:12:22.224119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:22.224066    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podStartSLOduration=4.102501786 podStartE2EDuration="6.22405353s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.121551744s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:22.222577235 +0000 UTC m=+2048.392934082" watchObservedRunningTime="2026-10-30 19:12:22.22405353 +0000 UTC m=+2048.394410377"
Oct 30 19:12:22.394090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:22.394008    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:23.939232 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:23.939197    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" path="/var/lib/kubelet/pods/eeb7aac9-3e03-47b0-a6ff-dde43bfa470a/volumes"
Oct 30 19:12:26.555979 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:26.555939    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:26.769230 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:26.769201    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:27.393944 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:27.393911    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:12:27.395444 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:27.395415    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 19:12:34.452824 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:34.452785    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:35.317240 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:35.317207    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:35.758498 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:35.758468    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:37.393558 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:37.393507    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 19:12:37.930669 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:37.930631    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:39.494143 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:39.494113    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:41.262346 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:41.262309    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:41.472616 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:41.472574    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:43.589360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:43.589324    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:45.835819 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:45.835784    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:47.394395 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:47.394355    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 19:12:49.872643 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:12:49.872598    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:12:57.394208 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:12:57.394156    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 19:13:00.117401 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:00.117372    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:00.326645 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:00.326611    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:01.769958 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:01.769929    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:03.136363 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:03.136327    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:07.393974 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:07.393935    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 19:13:17.393477 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.393430    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 19:13:17.727981 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.727907    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"]
Oct 30 19:13:17.728409 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.728391    4853 cpu_manager.go:414] "containerMap: removing container" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy"
Oct 30 19:13:17.728466 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.728413    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy"
Oct 30 19:13:17.728552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.728519    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy"
Oct 30 19:13:17.728594 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.728558    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy"
Oct 30 19:13:17.732929 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.732910    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.785006 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.784977    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"]
Oct 30 19:13:17.829409 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.829377    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.829409 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.829393    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Oct 30 19:13:17.829669 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.829429    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.829669 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.829449    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.829669 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.829586    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-cx4x8\""
Oct 30 19:13:17.829669 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.829611    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Oct 30 19:13:17.829669 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.829606    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.829669 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.829633    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Oct 30 19:13:17.829930 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.829668    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9rh\" (UniqueName: \"kubernetes.io/projected/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-kube-api-access-sx9rh\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.829930 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.829707    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.886204 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.886170    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:17.930469 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.930444    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.930642 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.930492    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.930642 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.930518    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.930763 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.930673    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.930763 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.930706    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9rh\" (UniqueName: \"kubernetes.io/projected/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-kube-api-access-sx9rh\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.930763 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.930745    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.930996 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.930972    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.931069 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.931029    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.931136 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.931068    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.932961 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.932934    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.933296 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.933272    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:17.986554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:17.986458    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9rh\" (UniqueName: \"kubernetes.io/projected/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-kube-api-access-sx9rh\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:18.048597 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:18.048566    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:18.343473 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:18.343442    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:18.417783 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:18.417749    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"]
Oct 30 19:13:18.426615 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:13:18.426587    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a402bf6_50ae_4dca_8ae2_e71d2eb8883d.slice/crio-f4e8b86ae14e0d06b3f12e96f8ae887a419b659970d54444b35ed0b28dda4939 WatchSource:0}: Error finding container f4e8b86ae14e0d06b3f12e96f8ae887a419b659970d54444b35ed0b28dda4939: Status 404 returned error can't find the container with id f4e8b86ae14e0d06b3f12e96f8ae887a419b659970d54444b35ed0b28dda4939
Oct 30 19:13:19.217615 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:19.217571    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks" event={"ID":"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d","Type":"ContainerStarted","Data":"62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6"}
Oct 30 19:13:19.217820 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:19.217673    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks" event={"ID":"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d","Type":"ContainerStarted","Data":"f4e8b86ae14e0d06b3f12e96f8ae887a419b659970d54444b35ed0b28dda4939"}
Oct 30 19:13:19.327156 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:19.327097    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks" podStartSLOduration=2.327085661 podStartE2EDuration="2.327085661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:19.326151787 +0000 UTC m=+2105.496508638" watchObservedRunningTime="2026-10-30 19:13:19.327085661 +0000 UTC m=+2105.497442508"
Oct 30 19:13:19.873905 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:13:19.873858    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:13:20.048999 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:20.048966    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:27.394270 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:27.394230    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 19:13:28.048699 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:28.048664    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:28.061348 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:28.061322    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:28.270274 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:28.270242    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:32.204063 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:32.203934    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:37.394216 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:37.394175    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 19:13:47.394205 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:47.394164    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 19:13:49.871735 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:13:49.871689    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:13:57.273718 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.273680    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"]
Oct 30 19:13:57.284701 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:13:57.284665    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6" cmd=["/bin/sleep","15"]
Oct 30 19:13:57.284859 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.284722    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:13:57.284859 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:13:57.284737    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks" podUID="6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" containerName="main" containerID="cri-o://62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6"
Oct 30 19:13:57.284859 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.284798    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks" podUID="6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" containerName="main" containerID="cri-o://62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6" gracePeriod=60
Oct 30 19:13:57.346750 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.346719    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:57.394741 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.394699    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 19:13:57.541298 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.541269    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:57.556131 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.556080    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:57.644386 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.644357    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-dshm\") pod \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") "
Oct 30 19:13:57.644609 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.644405    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tmp-dir\") pod \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") "
Oct 30 19:13:57.644609 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.644444    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-kube-api-access-sx9rh\" (UniqueName: \"kubernetes.io/projected/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-kube-api-access-sx9rh\") pod \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") "
Oct 30 19:13:57.644609 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.644473    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tls-certs\") pod \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") "
Oct 30 19:13:57.644609 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.644492    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-model-cache\") pod \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") "
Oct 30 19:13:57.644609 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.644562    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-home\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-home\") pod \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\" (UID: \"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d\") "
Oct 30 19:13:57.644896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.644734    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tmp-dir" pod "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" (UID: "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:57.644896 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.644827    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-model-cache" pod "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" (UID: "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:57.644994 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.644930    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-home" pod "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" (UID: "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:57.646826 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.646802    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-kube-api-access-sx9rh" pod "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" (UID: "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d"). InnerVolumeSpecName "kube-api-access-sx9rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:57.647209 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.647184    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-dshm" pod "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" (UID: "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:57.647346 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.647265    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tls-certs" pod "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" (UID: "6a402bf6-50ae-4dca-8ae2-e71d2eb8883d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:57.745362 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.745330    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:57.745362 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.745358    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:57.745362 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.745367    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:57.745362 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.745376    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9rh\" (UniqueName: \"kubernetes.io/projected/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-kube-api-access-sx9rh\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:57.745362 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.745386    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:57.745701 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:57.745396    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:58.398684 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:58.398645    4853 generic.go:356] "Generic (PLEG): container finished" podID="6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" containerID="62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6" exitCode=0
Oct 30 19:13:58.399084 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:58.398716    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"
Oct 30 19:13:58.399084 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:58.398739    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks" event={"ID":"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d","Type":"ContainerDied","Data":"62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6"}
Oct 30 19:13:58.399084 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:58.398793    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks" event={"ID":"6a402bf6-50ae-4dca-8ae2-e71d2eb8883d","Type":"ContainerDied","Data":"f4e8b86ae14e0d06b3f12e96f8ae887a419b659970d54444b35ed0b28dda4939"}
Oct 30 19:13:58.399084 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:58.398814    4853 scope.go:122] "RemoveContainer" containerID="62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6"
Oct 30 19:13:58.407697 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:58.407678    4853 scope.go:122] "RemoveContainer" containerID="62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6"
Oct 30 19:13:58.408003 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:13:58.407987    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6\": container with ID starting with 62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6 not found: ID does not exist" containerID="62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6"
Oct 30 19:13:58.408055 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:58.408010    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6"} err="failed to get container status \"62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6\": rpc error: code = NotFound desc = could not find container \"62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6\": container with ID starting with 62f80c6406f01539dcb56b208348b1f7196f0612a8660bc5b6fabc52948cfca6 not found: ID does not exist"
Oct 30 19:13:58.559474 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:58.559445    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"]
Oct 30 19:13:58.605827 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:58.605798    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777mqmks"]
Oct 30 19:13:59.939684 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:13:59.939637    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" path="/var/lib/kubelet/pods/6a402bf6-50ae-4dca-8ae2-e71d2eb8883d/volumes"
Oct 30 19:14:00.641874 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:00.641835    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:07.403483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:07.403444    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:14:07.411156 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:07.411128    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:14:15.511109 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:15.511077    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"]
Oct 30 19:14:15.586748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:15.586710    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:15.811876 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:15.811799    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:19.873845 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:14:19.873806    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:14:27.547758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:27.547725    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:30.533517 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:30.533476    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main" containerID="cri-o://4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752" gracePeriod=45
Oct 30 19:14:31.186745 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.186718    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:14:31.192754 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.192732    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a81bd8-b6d4-468d-bb5e-95db7854810f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a81bd8-b6d4-468d-bb5e-95db7854810f-cabundle-cert\") pod \"73a81bd8-b6d4-468d-bb5e-95db7854810f\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") "
Oct 30 19:14:31.192852 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.192777    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-home\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-home\") pod \"73a81bd8-b6d4-468d-bb5e-95db7854810f\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") "
Oct 30 19:14:31.192852 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.192802    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-tmp-dir\") pod \"73a81bd8-b6d4-468d-bb5e-95db7854810f\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") "
Oct 30 19:14:31.192852 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.192835    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-model-cache\") pod \"73a81bd8-b6d4-468d-bb5e-95db7854810f\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") "
Oct 30 19:14:31.193016 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.192871    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-kserve-provision-location\") pod \"73a81bd8-b6d4-468d-bb5e-95db7854810f\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") "
Oct 30 19:14:31.193016 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.192916    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a81bd8-b6d4-468d-bb5e-95db7854810f-tls-certs\" (UniqueName: \"kubernetes.io/secret/73a81bd8-b6d4-468d-bb5e-95db7854810f-tls-certs\") pod \"73a81bd8-b6d4-468d-bb5e-95db7854810f\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") "
Oct 30 19:14:31.193016 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.192943    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-dshm\") pod \"73a81bd8-b6d4-468d-bb5e-95db7854810f\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") "
Oct 30 19:14:31.193016 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.192971    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a81bd8-b6d4-468d-bb5e-95db7854810f-kube-api-access-pmxxx\" (UniqueName: \"kubernetes.io/projected/73a81bd8-b6d4-468d-bb5e-95db7854810f-kube-api-access-pmxxx\") pod \"73a81bd8-b6d4-468d-bb5e-95db7854810f\" (UID: \"73a81bd8-b6d4-468d-bb5e-95db7854810f\") "
Oct 30 19:14:31.193276 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.193084    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-model-cache" pod "73a81bd8-b6d4-468d-bb5e-95db7854810f" (UID: "73a81bd8-b6d4-468d-bb5e-95db7854810f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:31.193276 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.193148    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a81bd8-b6d4-468d-bb5e-95db7854810f-cabundle-cert" pod "73a81bd8-b6d4-468d-bb5e-95db7854810f" (UID: "73a81bd8-b6d4-468d-bb5e-95db7854810f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:31.193276 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.193231    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.193276 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.193248    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73a81bd8-b6d4-468d-bb5e-95db7854810f-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.193515 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.193494    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-home" pod "73a81bd8-b6d4-468d-bb5e-95db7854810f" (UID: "73a81bd8-b6d4-468d-bb5e-95db7854810f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:31.195321 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.195297    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a81bd8-b6d4-468d-bb5e-95db7854810f-kube-api-access-pmxxx" pod "73a81bd8-b6d4-468d-bb5e-95db7854810f" (UID: "73a81bd8-b6d4-468d-bb5e-95db7854810f"). InnerVolumeSpecName "kube-api-access-pmxxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:31.195417 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.195331    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-dshm" pod "73a81bd8-b6d4-468d-bb5e-95db7854810f" (UID: "73a81bd8-b6d4-468d-bb5e-95db7854810f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:31.195478 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.195458    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a81bd8-b6d4-468d-bb5e-95db7854810f-tls-certs" pod "73a81bd8-b6d4-468d-bb5e-95db7854810f" (UID: "73a81bd8-b6d4-468d-bb5e-95db7854810f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:31.205221 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.205196    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-tmp-dir" pod "73a81bd8-b6d4-468d-bb5e-95db7854810f" (UID: "73a81bd8-b6d4-468d-bb5e-95db7854810f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:31.220269 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.220250    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-kserve-provision-location" pod "73a81bd8-b6d4-468d-bb5e-95db7854810f" (UID: "73a81bd8-b6d4-468d-bb5e-95db7854810f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:31.293649 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.293598    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.293649 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.293618    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a81bd8-b6d4-468d-bb5e-95db7854810f-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.293649 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.293628    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.293649 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.293636    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmxxx\" (UniqueName: \"kubernetes.io/projected/73a81bd8-b6d4-468d-bb5e-95db7854810f-kube-api-access-pmxxx\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.293649 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.293644    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.293859 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.293653    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a81bd8-b6d4-468d-bb5e-95db7854810f-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.546075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.546014    4853 generic.go:356] "Generic (PLEG): container finished" podID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerID="4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752" exitCode=0
Oct 30 19:14:31.546409 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.546066    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" event={"ID":"73a81bd8-b6d4-468d-bb5e-95db7854810f","Type":"ContainerDied","Data":"4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752"}
Oct 30 19:14:31.546409 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.546122    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n" event={"ID":"73a81bd8-b6d4-468d-bb5e-95db7854810f","Type":"ContainerDied","Data":"cf77f54c7138be853a9bf2709e26b5cb58404b18497c44d823f82b4ae4b8b3b5"}
Oct 30 19:14:31.546409 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.546144    4853 scope.go:122] "RemoveContainer" containerID="4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752"
Oct 30 19:14:31.546409 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.546077    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"
Oct 30 19:14:31.554708 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.554687    4853 scope.go:122] "RemoveContainer" containerID="a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8"
Oct 30 19:14:31.562069 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.562050    4853 scope.go:122] "RemoveContainer" containerID="4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752"
Oct 30 19:14:31.562315 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:14:31.562297    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752\": container with ID starting with 4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752 not found: ID does not exist" containerID="4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752"
Oct 30 19:14:31.562360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.562326    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752"} err="failed to get container status \"4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752\": rpc error: code = NotFound desc = could not find container \"4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752\": container with ID starting with 4fd8327f2b26e1d27a6225e3645973ec794ebea7a4c0951affaad1916d428752 not found: ID does not exist"
Oct 30 19:14:31.562360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.562343    4853 scope.go:122] "RemoveContainer" containerID="a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8"
Oct 30 19:14:31.562597 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:14:31.562571    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8\": container with ID starting with a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8 not found: ID does not exist" containerID="a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8"
Oct 30 19:14:31.562695 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.562603    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8"} err="failed to get container status \"a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8\": rpc error: code = NotFound desc = could not find container \"a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8\": container with ID starting with a6ee8399b95de9f70d45cb9325dc36ead393eb1ec0bed68678915e3a9bac4ad8 not found: ID does not exist"
Oct 30 19:14:31.703805 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.703780    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"]
Oct 30 19:14:31.709748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.709729    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b77c4cb55w7g9n"]
Oct 30 19:14:31.944329 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:31.944296    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" path="/var/lib/kubelet/pods/73a81bd8-b6d4-468d-bb5e-95db7854810f/volumes"
Oct 30 19:14:46.586363 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:46.586332    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:46.799818 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:14:46.799749    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:49.875401 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:14:49.875360    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:15:00.926849 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:00.926814    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:05.126016 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:05.125981    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:05.335190 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:05.335161    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:12.283337 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.283304    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"]
Oct 30 19:15:12.283748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.283588    4853 cpu_manager.go:414] "containerMap: removing container" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="storage-initializer"
Oct 30 19:15:12.283748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.283598    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="storage-initializer"
Oct 30 19:15:12.283748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.283614    4853 cpu_manager.go:414] "containerMap: removing container" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy"
Oct 30 19:15:12.283748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.283618    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb7aac9-3e03-47b0-a6ff-dde43bfa470a" containerName="istio-proxy"
Oct 30 19:15:12.283748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.283631    4853 cpu_manager.go:414] "containerMap: removing container" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main"
Oct 30 19:15:12.283748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.283637    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main"
Oct 30 19:15:12.283748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.283653    4853 cpu_manager.go:414] "containerMap: removing container" podUID="6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" containerName="main"
Oct 30 19:15:12.283748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.283657    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" containerName="main"
Oct 30 19:15:12.283748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.283715    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a81bd8-b6d4-468d-bb5e-95db7854810f" containerName="main"
Oct 30 19:15:12.283748 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.283724    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a402bf6-50ae-4dca-8ae2-e71d2eb8883d" containerName="main"
Oct 30 19:15:12.286868 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.286848    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.331885 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.331858    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"]
Oct 30 19:15:12.338664 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.338631    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:12.367773 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.367745    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.367871 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.367780    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.367871 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.367797    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.367871 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.367859    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4393d86-1b2a-46dc-a0bf-976893882527-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.368018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.367931    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.368018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.367949    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.368122 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.368016    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455w5\" (UniqueName: \"kubernetes.io/projected/f4393d86-1b2a-46dc-a0bf-976893882527-kube-api-access-455w5\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.376893 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.376871    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.376893 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.376874    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.377005 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.376935    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-2278p\""
Oct 30 19:15:12.377005 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.376969    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Oct 30 19:15:12.469193 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.469170    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.469313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.469202    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.469313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.469219    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.469313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.469244    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4393d86-1b2a-46dc-a0bf-976893882527-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.469313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.469298    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.469522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.469321    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.469522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.469344    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455w5\" (UniqueName: \"kubernetes.io/projected/f4393d86-1b2a-46dc-a0bf-976893882527-kube-api-access-455w5\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.469660 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.469644    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.469759 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.469737    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.469820 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.469758    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.469820 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.469785    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.471844 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.471821    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.472088 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.472073    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4393d86-1b2a-46dc-a0bf-976893882527-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.535302 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.535252    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-455w5\" (UniqueName: \"kubernetes.io/projected/f4393d86-1b2a-46dc-a0bf-976893882527-kube-api-access-455w5\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:12.603376 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:12.603352    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:13.001931 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:13.001899    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:13.073712 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:13.073679    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"]
Oct 30 19:15:13.118883 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:13.118864    4853 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:15:13.238444 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:13.238417    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:13.738162 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:13.738123    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" event={"ID":"f4393d86-1b2a-46dc-a0bf-976893882527","Type":"ContainerStarted","Data":"7d70147826d9a41845d68cc42be053957c57e21aeb337b9615376287e7166ed6"}
Oct 30 19:15:16.635975 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:16.635940    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:19.696251 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:19.696218    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:19.873399 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:15:19.873365    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:15:19.915774 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:19.915740    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:32.825021 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:32.824955    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" event={"ID":"f4393d86-1b2a-46dc-a0bf-976893882527","Type":"ContainerStarted","Data":"ffa2a8c9f8e4b11dde5aabd0810a73e69aab9c9fb70a2b03732299d34400d6b2"}
Oct 30 19:15:34.834938 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:34.834903    4853 generic.go:356] "Generic (PLEG): container finished" podID="f4393d86-1b2a-46dc-a0bf-976893882527" containerID="ffa2a8c9f8e4b11dde5aabd0810a73e69aab9c9fb70a2b03732299d34400d6b2" exitCode=0
Oct 30 19:15:34.834938 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:34.834940    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" event={"ID":"f4393d86-1b2a-46dc-a0bf-976893882527","Type":"ContainerDied","Data":"ffa2a8c9f8e4b11dde5aabd0810a73e69aab9c9fb70a2b03732299d34400d6b2"}
Oct 30 19:15:35.843122 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:35.843091    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" event={"ID":"f4393d86-1b2a-46dc-a0bf-976893882527","Type":"ContainerStarted","Data":"555a79ce3f77e6c5ff769d6d41ffa979d01872e8695de6452d1cd1de319b8cc9"}
Oct 30 19:15:36.212434 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:36.212404    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:36.433940 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:36.433912    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:36.850882 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:36.850848    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" event={"ID":"f4393d86-1b2a-46dc-a0bf-976893882527","Type":"ContainerStarted","Data":"d3fec852f5251084f2b2b35b4ad4369fabe588f6c6c4eeaa8f37b92f6797cb37"}
Oct 30 19:15:37.058029 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:37.057963    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podStartSLOduration=3.72247068 podStartE2EDuration="25.057944369s" totalImagesPullingTime="20.049905601s" totalInitContainerRuntime="1.285568088s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:37.056038144 +0000 UTC m=+2243.226394992" watchObservedRunningTime="2026-10-30 19:15:37.057944369 +0000 UTC m=+2243.228301221"
Oct 30 19:15:37.604279 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:37.604248    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:42.604317 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:42.604287    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:15:42.605909 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:42.605881    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 19:15:44.666457 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:44.666430    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:48.768277 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:48.768237    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:49.234177 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:49.234139    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:49.874362 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:15:49.874324    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:15:52.378583 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:52.378540    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:52.604279 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:52.604227    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 19:15:52.605008 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:15:52.604987    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:01.328855 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:16:01.328822    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:01.543849 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:16:01.543812    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:02.604079 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:16:02.604033    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 19:16:04.378922 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:16:04.378878    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:12.604062 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:16:12.604025    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 19:16:19.872481 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:16:19.872443    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:16:22.603917 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:16:22.603876    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 19:16:32.603725 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:16:32.603685    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 19:16:42.604242 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:16:42.604197    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 19:16:46.408347 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:16:46.408302    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:46.619957 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:16:46.619911    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:49.874495 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:16:49.874453    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:16:52.604706 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:16:52.604668    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 19:17:01.661585 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:01.661553    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:02.604133 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:02.604086    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 19:17:12.604753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:12.604702    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 19:17:19.873076 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:17:19.873038    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:17:22.614109 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:22.614071    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:17:22.623422 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:22.623390    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:17:30.406039 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:30.406008    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"]
Oct 30 19:17:30.406480 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:30.406327    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="modelcar" containerID="cri-o://d3fec852f5251084f2b2b35b4ad4369fabe588f6c6c4eeaa8f37b92f6797cb37" gracePeriod=60
Oct 30 19:17:30.506358 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:30.506300    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:30.718401 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:30.718324    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:31.374386 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:31.374356    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp_f4393d86-1b2a-46dc-a0bf-976893882527/modelcar/0.log"
Oct 30 19:17:31.375029 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:31.375007    4853 generic.go:356] "Generic (PLEG): container finished" podID="f4393d86-1b2a-46dc-a0bf-976893882527" containerID="d3fec852f5251084f2b2b35b4ad4369fabe588f6c6c4eeaa8f37b92f6797cb37" exitCode=143
Oct 30 19:17:31.375141 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:31.375083    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" event={"ID":"f4393d86-1b2a-46dc-a0bf-976893882527","Type":"ContainerDied","Data":"d3fec852f5251084f2b2b35b4ad4369fabe588f6c6c4eeaa8f37b92f6797cb37"}
Oct 30 19:17:37.721599 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:37.721564    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:37.937076 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:37.937035    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:45.429077 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:45.429002    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" containerID="cri-o://555a79ce3f77e6c5ff769d6d41ffa979d01872e8695de6452d1cd1de319b8cc9" gracePeriod=45
Oct 30 19:17:45.615773 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:45.615729    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 19:17:46.446096 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.446071    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp_f4393d86-1b2a-46dc-a0bf-976893882527/modelcar/0.log"
Oct 30 19:17:46.447023 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.446973    4853 generic.go:356] "Generic (PLEG): container finished" podID="f4393d86-1b2a-46dc-a0bf-976893882527" containerID="555a79ce3f77e6c5ff769d6d41ffa979d01872e8695de6452d1cd1de319b8cc9" exitCode=0
Oct 30 19:17:46.447162 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.447054    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" event={"ID":"f4393d86-1b2a-46dc-a0bf-976893882527","Type":"ContainerDied","Data":"555a79ce3f77e6c5ff769d6d41ffa979d01872e8695de6452d1cd1de319b8cc9"}
Oct 30 19:17:46.670027 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.670006    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp_f4393d86-1b2a-46dc-a0bf-976893882527/modelcar/0.log"
Oct 30 19:17:46.671828 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.671811    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:17:46.799494 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.799439    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-home\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-home\") pod \"f4393d86-1b2a-46dc-a0bf-976893882527\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") "
Oct 30 19:17:46.799494 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.799468    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-tmp-dir\") pod \"f4393d86-1b2a-46dc-a0bf-976893882527\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") "
Oct 30 19:17:46.799663 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.799511    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4393d86-1b2a-46dc-a0bf-976893882527-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4393d86-1b2a-46dc-a0bf-976893882527-tls-certs\") pod \"f4393d86-1b2a-46dc-a0bf-976893882527\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") "
Oct 30 19:17:46.799663 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.799566    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-model-cache\") pod \"f4393d86-1b2a-46dc-a0bf-976893882527\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") "
Oct 30 19:17:46.799663 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.799607    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-kserve-provision-location\") pod \"f4393d86-1b2a-46dc-a0bf-976893882527\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") "
Oct 30 19:17:46.799663 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.799653    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-dshm\") pod \"f4393d86-1b2a-46dc-a0bf-976893882527\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") "
Oct 30 19:17:46.799878 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.799690    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4393d86-1b2a-46dc-a0bf-976893882527-kube-api-access-455w5\" (UniqueName: \"kubernetes.io/projected/f4393d86-1b2a-46dc-a0bf-976893882527-kube-api-access-455w5\") pod \"f4393d86-1b2a-46dc-a0bf-976893882527\" (UID: \"f4393d86-1b2a-46dc-a0bf-976893882527\") "
Oct 30 19:17:46.799936 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.799859    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-model-cache" pod "f4393d86-1b2a-46dc-a0bf-976893882527" (UID: "f4393d86-1b2a-46dc-a0bf-976893882527"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:46.800022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.799990    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-kserve-provision-location" pod "f4393d86-1b2a-46dc-a0bf-976893882527" (UID: "f4393d86-1b2a-46dc-a0bf-976893882527"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:46.800290 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.800252    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-home" pod "f4393d86-1b2a-46dc-a0bf-976893882527" (UID: "f4393d86-1b2a-46dc-a0bf-976893882527"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:46.801843 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.801823    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4393d86-1b2a-46dc-a0bf-976893882527-tls-certs" pod "f4393d86-1b2a-46dc-a0bf-976893882527" (UID: "f4393d86-1b2a-46dc-a0bf-976893882527"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:46.802043 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.802016    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-dshm" pod "f4393d86-1b2a-46dc-a0bf-976893882527" (UID: "f4393d86-1b2a-46dc-a0bf-976893882527"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:46.802427 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.802409    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4393d86-1b2a-46dc-a0bf-976893882527-kube-api-access-455w5" pod "f4393d86-1b2a-46dc-a0bf-976893882527" (UID: "f4393d86-1b2a-46dc-a0bf-976893882527"). InnerVolumeSpecName "kube-api-access-455w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:46.811258 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.811235    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-tmp-dir" pod "f4393d86-1b2a-46dc-a0bf-976893882527" (UID: "f4393d86-1b2a-46dc-a0bf-976893882527"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:46.900163 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.900140    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4393d86-1b2a-46dc-a0bf-976893882527-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.900163 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.900160    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.900292 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.900171    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.900292 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.900189    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.900292 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.900198    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-455w5\" (UniqueName: \"kubernetes.io/projected/f4393d86-1b2a-46dc-a0bf-976893882527-kube-api-access-455w5\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.900292 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.900206    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:46.900292 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:46.900214    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4393d86-1b2a-46dc-a0bf-976893882527-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:47.453941 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:47.453914    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp_f4393d86-1b2a-46dc-a0bf-976893882527/modelcar/0.log"
Oct 30 19:17:47.454850 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:47.454823    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" event={"ID":"f4393d86-1b2a-46dc-a0bf-976893882527","Type":"ContainerDied","Data":"7d70147826d9a41845d68cc42be053957c57e21aeb337b9615376287e7166ed6"}
Oct 30 19:17:47.454921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:47.454896    4853 scope.go:122] "RemoveContainer" containerID="d3fec852f5251084f2b2b35b4ad4369fabe588f6c6c4eeaa8f37b92f6797cb37"
Oct 30 19:17:47.454968 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:47.454835    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:17:47.465942 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:47.465922    4853 scope.go:122] "RemoveContainer" containerID="555a79ce3f77e6c5ff769d6d41ffa979d01872e8695de6452d1cd1de319b8cc9"
Oct 30 19:17:47.474263 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:47.474244    4853 scope.go:122] "RemoveContainer" containerID="ffa2a8c9f8e4b11dde5aabd0810a73e69aab9c9fb70a2b03732299d34400d6b2"
Oct 30 19:17:47.615265 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:47.615234    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Oct 30 19:17:47.615401 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:47.615323    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"
Oct 30 19:17:47.660958 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:47.660929    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"]
Oct 30 19:17:47.707727 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:47.707653    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bzgkrp"]
Oct 30 19:17:47.939860 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:17:47.939814    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" path="/var/lib/kubelet/pods/f4393d86-1b2a-46dc-a0bf-976893882527/volumes"
Oct 30 19:17:49.875818 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:17:49.875776    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:18:02.032254 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:02.032206    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:19.219854 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:19.219815    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:19.876001 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:18:19.875963    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:18:20.451072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:20.451037    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:25.594008 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.593970    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"]
Oct 30 19:18:25.594555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.594513    4853 cpu_manager.go:414] "containerMap: removing container" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main"
Oct 30 19:18:25.594627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.594561    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main"
Oct 30 19:18:25.594627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.594576    4853 cpu_manager.go:414] "containerMap: removing container" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="modelcar-init"
Oct 30 19:18:25.594627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.594583    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="modelcar-init"
Oct 30 19:18:25.594627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.594611    4853 cpu_manager.go:414] "containerMap: removing container" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="modelcar"
Oct 30 19:18:25.594627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.594617    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="modelcar"
Oct 30 19:18:25.594808 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.594684    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="main"
Oct 30 19:18:25.594808 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.594695    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4393d86-1b2a-46dc-a0bf-976893882527" containerName="modelcar"
Oct 30 19:18:25.599039 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.599016    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.642502 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.642476    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"]
Oct 30 19:18:25.653960 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.653931    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.654072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.653973    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8471ed21-b031-4f51-92f1-9ad6f069eae2-tls-certs\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.654072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.654035    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-dshm\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.654072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.654067    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-tmp-dir\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.654201 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.654083    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8471ed21-b031-4f51-92f1-9ad6f069eae2-cabundle-cert\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.654201 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.654109    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-home\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.654201 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.654160    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-model-cache\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.654305 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.654266    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhnl\" (UniqueName: \"kubernetes.io/projected/8471ed21-b031-4f51-92f1-9ad6f069eae2-kube-api-access-wrhnl\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.687342 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.687285    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-k76hw\""
Oct 30 19:18:25.687342 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.687285    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Oct 30 19:18:25.687342 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.687292    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Oct 30 19:18:25.687342 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.687318    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Oct 30 19:18:25.687342 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.687340    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:18:25.687689 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.687318    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Oct 30 19:18:25.755034 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755013    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8471ed21-b031-4f51-92f1-9ad6f069eae2-tls-certs\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755136 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755044    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-dshm\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755136 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755072    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-tmp-dir\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755239 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755213    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8471ed21-b031-4f51-92f1-9ad6f069eae2-cabundle-cert\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755294 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755279    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-home\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755340 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755313    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-model-cache\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755340 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755333    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhnl\" (UniqueName: \"kubernetes.io/projected/8471ed21-b031-4f51-92f1-9ad6f069eae2-kube-api-access-wrhnl\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755421 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755362    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755467 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755446    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-tmp-dir\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755715 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755673    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-home\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755849 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755735    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-model-cache\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755849 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755772    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.755978 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.755898    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8471ed21-b031-4f51-92f1-9ad6f069eae2-cabundle-cert\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.757653 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.757633    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-dshm\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.758097 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.758082    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8471ed21-b031-4f51-92f1-9ad6f069eae2-tls-certs\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.819138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.819103    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhnl\" (UniqueName: \"kubernetes.io/projected/8471ed21-b031-4f51-92f1-9ad6f069eae2-kube-api-access-wrhnl\") pod \"custom-route-timeout-test-kserve-d97cf856-zfdxb\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:25.913985 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:25.913928    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:26.340344 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:26.340302    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"]
Oct 30 19:18:26.347141 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:18:26.347103    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8471ed21_b031_4f51_92f1_9ad6f069eae2.slice/crio-f352753e308c835a9292ea3216cc1283d4af7de6092e5de38f1a6a67c2aabc9f WatchSource:0}: Error finding container f352753e308c835a9292ea3216cc1283d4af7de6092e5de38f1a6a67c2aabc9f: Status 404 returned error can't find the container with id f352753e308c835a9292ea3216cc1283d4af7de6092e5de38f1a6a67c2aabc9f
Oct 30 19:18:26.633443 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:26.633352    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" event={"ID":"8471ed21-b031-4f51-92f1-9ad6f069eae2","Type":"ContainerStarted","Data":"a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca"}
Oct 30 19:18:26.633443 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:26.633414    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" event={"ID":"8471ed21-b031-4f51-92f1-9ad6f069eae2","Type":"ContainerStarted","Data":"f352753e308c835a9292ea3216cc1283d4af7de6092e5de38f1a6a67c2aabc9f"}
Oct 30 19:18:27.222455 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:27.222416    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:27.448686 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:27.448646    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:28.283428 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:28.283392    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:28.645955 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:28.645928    4853 generic.go:356] "Generic (PLEG): container finished" podID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerID="a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca" exitCode=0
Oct 30 19:18:28.646100 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:28.645984    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" event={"ID":"8471ed21-b031-4f51-92f1-9ad6f069eae2","Type":"ContainerDied","Data":"a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca"}
Oct 30 19:18:29.652968 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:29.652933    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" event={"ID":"8471ed21-b031-4f51-92f1-9ad6f069eae2","Type":"ContainerStarted","Data":"e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753"}
Oct 30 19:18:29.859974 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:29.859905    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podStartSLOduration=2.706185471 podStartE2EDuration="4.859885523s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.153700052s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:29.85956079 +0000 UTC m=+2416.029917635" watchObservedRunningTime="2026-10-30 19:18:29.859885523 +0000 UTC m=+2416.030242368"
Oct 30 19:18:29.914235 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:29.914165    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:34.858375 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:34.858338    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:35.704332 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:35.704295    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:35.914974 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:35.914933    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:18:35.916635 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:35.916610    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 19:18:35.928955 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:35.928932    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:41.432826 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:41.432793    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:41.646793 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:41.646757    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:43.737776 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:43.737739    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:44.239812 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:44.239780    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:45.914943 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:45.914902    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 19:18:49.572931 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:49.572895    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:49.873318 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:18:49.873280    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:18:50.041480 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:50.041445    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:50.372895 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:50.372858    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:53.509576 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:53.509519    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:55.914316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:18:55.914272    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 19:19:00.633833 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.633801    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"]
Oct 30 19:19:00.636095 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.636079    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.680469 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.680437    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"]
Oct 30 19:19:00.682662 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.682641    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.701031 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.701010    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-dstgc\""
Oct 30 19:19:00.715327 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715294    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.715443 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715341    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.715443 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715421    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.715588 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715464    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.715588 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715505    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60c4763-4f32-48c7-8063-e977053d11d4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.715588 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715554    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.715588 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715579    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.715789 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715605    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lswht\" (UniqueName: \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-kube-api-access-lswht\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.715789 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715632    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.715789 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715659    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.715789 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715680    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.715789 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715709    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60c4763-4f32-48c7-8063-e977053d11d4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.715789 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715732    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.715789 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715758    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgjl\" (UniqueName: \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-kube-api-access-2sgjl\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.715789 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715783    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.716136 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715806    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.716136 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.715975    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.716136 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.716060    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.731376 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.731355    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"]
Oct 30 19:19:00.779140 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.779101    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"]
Oct 30 19:19:00.817127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817098    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.817127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817130    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.817350 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817155    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.817350 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817185    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.817350 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817229    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.817350 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817266    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.817350 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817306    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60c4763-4f32-48c7-8063-e977053d11d4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.817350 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817335    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.817698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817364    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.817698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817392    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lswht\" (UniqueName: \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-kube-api-access-lswht\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.817698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817425    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.817698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817453    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.817698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817478    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.817698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817510    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60c4763-4f32-48c7-8063-e977053d11d4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.817698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817558    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.817698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817587    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgjl\" (UniqueName: \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-kube-api-access-2sgjl\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.817698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817613    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.817698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817635    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.817698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817650    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.818483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817705    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.818483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817923    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.818483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.817986    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.818483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.818070    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.818483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.818388    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.818780 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.818500    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.818780 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.818517    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60c4763-4f32-48c7-8063-e977053d11d4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.818971 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.818949    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.819167 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.819140    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.820185 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.820160    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.820506 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.820482    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60c4763-4f32-48c7-8063-e977053d11d4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.820654 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.820636    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.820839 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.820823    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.877670 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.877620    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.877961 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.877939    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgjl\" (UniqueName: \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-kube-api-access-2sgjl\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.878105 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.878085    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lswht\" (UniqueName: \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-kube-api-access-lswht\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-phjn7\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:00.878429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.878410    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nxn8z\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.947719 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.947661    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:00.994351 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:00.994276    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:01.372353 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.372316    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"]
Oct 30 19:19:01.379032 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:19:01.379006    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7713d2_baf3_4ac9_b046_9c944e5f629e.slice/crio-0778175c5caeaeccafe9a0cab92250acbdafcd4deb0f8c169523a456ea1d01fd WatchSource:0}: Error finding container 0778175c5caeaeccafe9a0cab92250acbdafcd4deb0f8c169523a456ea1d01fd: Status 404 returned error can't find the container with id 0778175c5caeaeccafe9a0cab92250acbdafcd4deb0f8c169523a456ea1d01fd
Oct 30 19:19:01.381188 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.381150    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:01.381297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.381236    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:01.381297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.381282    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:01.427987 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.427968    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"]
Oct 30 19:19:01.434480 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:19:01.434452    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60c4763_4f32_48c7_8063_e977053d11d4.slice/crio-8cb5597456c92c5c0d6514ef9a3ba51f0decd7951bbe56558b6871a673657dc1 WatchSource:0}: Error finding container 8cb5597456c92c5c0d6514ef9a3ba51f0decd7951bbe56558b6871a673657dc1: Status 404 returned error can't find the container with id 8cb5597456c92c5c0d6514ef9a3ba51f0decd7951bbe56558b6871a673657dc1
Oct 30 19:19:01.436491 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.436463    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:01.436572 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.436554    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:01.436619 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.436584    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:01.802365 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.802264    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" event={"ID":"f60c4763-4f32-48c7-8063-e977053d11d4","Type":"ContainerStarted","Data":"4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6"}
Oct 30 19:19:01.802365 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.802328    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" event={"ID":"f60c4763-4f32-48c7-8063-e977053d11d4","Type":"ContainerStarted","Data":"8cb5597456c92c5c0d6514ef9a3ba51f0decd7951bbe56558b6871a673657dc1"}
Oct 30 19:19:01.804105 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.804080    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" event={"ID":"ed7713d2-baf3-4ac9-b046-9c944e5f629e","Type":"ContainerStarted","Data":"da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00"}
Oct 30 19:19:01.804224 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.804128    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" event={"ID":"ed7713d2-baf3-4ac9-b046-9c944e5f629e","Type":"ContainerStarted","Data":"0778175c5caeaeccafe9a0cab92250acbdafcd4deb0f8c169523a456ea1d01fd"}
Oct 30 19:19:01.919158 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.919086    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" podStartSLOduration=1.9190682940000001 podStartE2EDuration="1.919068294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:01.91783524 +0000 UTC m=+2448.088192089" watchObservedRunningTime="2026-10-30 19:19:01.919068294 +0000 UTC m=+2448.089425141"
Oct 30 19:19:01.956554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.956003    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:01.994960 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:01.994921    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:02.043554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:02.043190    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" podStartSLOduration=2.043171258 podStartE2EDuration="2.043171258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:02.041130484 +0000 UTC m=+2448.211487333" watchObservedRunningTime="2026-10-30 19:19:02.043171258 +0000 UTC m=+2448.213528109"
Oct 30 19:19:02.343565 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:02.341839    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:02.550554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:02.547864    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:02.955940 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:02.955621    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:19:02.955940 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:02.955680    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" podUID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:19:02.995935 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:02.995907    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-phjn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:19:02.996086 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:02.995958    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" podUID="f60c4763-4f32-48c7-8063-e977053d11d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 19:19:03.951555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:03.950233    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:19:03.951555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:03.950287    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" podUID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:19:03.994925 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:03.994878    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-phjn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:19:03.995431 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:03.994941    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" podUID="f60c4763-4f32-48c7-8063-e977053d11d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 19:19:04.954554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:04.950842    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:19:04.954554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:04.950915    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" podUID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:19:04.995258 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:04.995221    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-phjn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:19:04.995669 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:04.995275    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" podUID="f60c4763-4f32-48c7-8063-e977053d11d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 19:19:05.914667 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:05.914619    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 19:19:05.952800 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:05.952764    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:19:05.953057 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:05.953029    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" podUID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:19:05.995582 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:05.995544    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-phjn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:19:05.995980 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:05.995607    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" podUID="f60c4763-4f32-48c7-8063-e977053d11d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 19:19:06.950178 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:06.950140    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:19:06.950451 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:06.950422    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" podUID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 19:19:06.994727 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:06.994690    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-phjn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:19:06.994876 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:06.994746    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" podUID="f60c4763-4f32-48c7-8063-e977053d11d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 19:19:07.952016 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:07.951987    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:07.952508 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:07.952349    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:07.953081 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:07.953065    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:19:07.998800 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:07.998778    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:07.999049 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:07.999033    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:07.999708 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:07.999687    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:19:15.754591 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.754556    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"]
Oct 30 19:19:15.758426 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.758383    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.801172 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.801143    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"]
Oct 30 19:19:15.803318 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.803303    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.852671 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.852635    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"]
Oct 30 19:19:15.873157 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873117    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.873157 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873158    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.873372 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873232    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.873372 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873259    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.873372 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873328    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.873372 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873360    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.873624 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873403    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.873624 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873481    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.873624 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873514    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.873624 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873555    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.873624 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873599    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.873624 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873617    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.873844 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873646    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nctls\" (UniqueName: \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-kube-api-access-nctls\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.873844 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873665    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6lg\" (UniqueName: \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-kube-api-access-pt6lg\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.873844 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873685    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.873844 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873809    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee75a04-5218-4428-b5f2-eb0403792f33-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.873994 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873858    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.873994 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.873901    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.907717 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.907688    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"]
Oct 30 19:19:15.914751 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.914722    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 19:19:15.974756 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.974725    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.974945 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.974771    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.974945 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.974911    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.975231 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975173    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.975374 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975353    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.975441 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975399    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.975480 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975443    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.975480 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975466    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.975596 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975495    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.975596 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975519    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.975776 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975712    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nctls\" (UniqueName: \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-kube-api-access-nctls\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.975844 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975794    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6lg\" (UniqueName: \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-kube-api-access-pt6lg\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.975844 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975825    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.975947 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975862    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee75a04-5218-4428-b5f2-eb0403792f33-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.975947 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975918    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.976048 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975949    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.976048 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.975982    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.976048 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.976012    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.976203 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.976051    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.976203 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.976056    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.976203 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.976111    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.976359 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.976331    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.976623 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.976601    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.976711 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.976694    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.976766 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.976727    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.976973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.976947    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.977061 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.977012    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee75a04-5218-4428-b5f2-eb0403792f33-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.977584 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.977506    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.978032 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.977758    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.979327 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.979302    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:15.979483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.979462    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:15.980652 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:15.980234    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:16.031731 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.031593    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:16.032660 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.032640    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nctls\" (UniqueName: \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-kube-api-access-nctls\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:16.032973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.032951    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-zskzv\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:16.033379 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.033363    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6lg\" (UniqueName: \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-kube-api-access-pt6lg\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-m7s9f\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:16.072913 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.072893    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:16.121205 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.121171    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:16.519181 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.519151    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"]
Oct 30 19:19:16.525749 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:19:16.525718    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee75a04_5218_4428_b5f2_eb0403792f33.slice/crio-3dff2c767629ddf9ddc78d5efb241a9ed18c2e22eb5b9a7ee9e0e3966ae57c37 WatchSource:0}: Error finding container 3dff2c767629ddf9ddc78d5efb241a9ed18c2e22eb5b9a7ee9e0e3966ae57c37: Status 404 returned error can't find the container with id 3dff2c767629ddf9ddc78d5efb241a9ed18c2e22eb5b9a7ee9e0e3966ae57c37
Oct 30 19:19:16.527774 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.527741    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:16.527863 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.527807    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:16.527863 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.527833    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:16.573361 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.573337    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"]
Oct 30 19:19:16.579342 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:19:16.579317    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf075728f_b71d_4b4d_82ea_3d10ec3b80c7.slice/crio-daec5f9f5649409a3ae5c2ea52a822d5ec19e78eecb485c955032391df275986 WatchSource:0}: Error finding container daec5f9f5649409a3ae5c2ea52a822d5ec19e78eecb485c955032391df275986: Status 404 returned error can't find the container with id daec5f9f5649409a3ae5c2ea52a822d5ec19e78eecb485c955032391df275986
Oct 30 19:19:16.581401 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.581373    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:16.581476 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.581432    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:16.581476 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.581464    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:19:16.880381 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.880342    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" event={"ID":"0ee75a04-5218-4428-b5f2-eb0403792f33","Type":"ContainerStarted","Data":"7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a"}
Oct 30 19:19:16.880920 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.880409    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" event={"ID":"0ee75a04-5218-4428-b5f2-eb0403792f33","Type":"ContainerStarted","Data":"3dff2c767629ddf9ddc78d5efb241a9ed18c2e22eb5b9a7ee9e0e3966ae57c37"}
Oct 30 19:19:16.882781 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.882752    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" event={"ID":"f075728f-b71d-4b4d-82ea-3d10ec3b80c7","Type":"ContainerStarted","Data":"20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925"}
Oct 30 19:19:16.882920 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.882799    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" event={"ID":"f075728f-b71d-4b4d-82ea-3d10ec3b80c7","Type":"ContainerStarted","Data":"daec5f9f5649409a3ae5c2ea52a822d5ec19e78eecb485c955032391df275986"}
Oct 30 19:19:16.997170 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:16.997109    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" podStartSLOduration=1.997090561 podStartE2EDuration="1.997090561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:16.995328077 +0000 UTC m=+2463.165684925" watchObservedRunningTime="2026-10-30 19:19:16.997090561 +0000 UTC m=+2463.167447410"
Oct 30 19:19:17.081446 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:17.081008    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:17.081446 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:17.081131    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-zskzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:19:17.081446 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:17.081174    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:19:17.115479 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:17.115407    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" podStartSLOduration=2.115384341 podStartE2EDuration="2.115384341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:17.112862543 +0000 UTC m=+2463.283219412" watchObservedRunningTime="2026-10-30 19:19:17.115384341 +0000 UTC m=+2463.285741197"
Oct 30 19:19:17.122027 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:17.121992    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:17.123498 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:17.123470    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:19:17.123664 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:17.123522    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:19:18.073979 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:18.073943    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-zskzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:19:18.074611 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:18.074578    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:19:18.122227 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:18.122200    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:19:18.122399 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:18.122382    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:19:19.079424 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:19.078966    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-zskzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:19:19.079424 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:19.079038    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:19:19.122270 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:19.122229    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:19:19.122446 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:19.122300    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:19:19.911492 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:19:19.911450    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:19:20.074341 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:20.073902    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-zskzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:19:20.074341 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:20.073963    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:19:20.122212 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:20.122169    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:19:20.122603 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:20.122230    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:19:21.077126 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:21.076933    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-zskzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:19:21.077126 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:21.076999    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:19:21.122092 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:21.122055    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:19:21.122251 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:21.122107    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:19:22.076047 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:22.076011    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-zskzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:19:22.076846 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:22.076343    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 19:19:22.122549 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:22.122500    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:19:22.122965 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:22.122583    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 19:19:23.077730 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:23.077701    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:23.077986 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:23.077964    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:23.078705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:23.078683    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:19:23.126287 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:23.126260    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:23.126767 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:23.126550    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:23.127081 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:23.127063    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:19:25.914885 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:25.914835    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 19:19:31.575717 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:31.575683    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:31.824246 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:31.824205    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:35.914308 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:35.914260    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 19:19:45.914304 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:45.914251    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 19:19:49.873945 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:19:49.873905    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:19:55.915056 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:19:55.915002    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 19:20:02.723799 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:02.723760    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:05.915341 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:05.915296    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 19:20:10.769042 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:10.769006    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:11.046955 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:11.046866    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:15.915290 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:15.915251    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 19:20:19.873556 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:20:19.873494    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:20:25.924915 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:25.924882    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:20:25.932920 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:25.932888    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:20:33.962459 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:33.962428    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"]
Oct 30 19:20:34.044332 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:34.044303    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:34.255316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:34.255235    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:48.985276 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:48.985218    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main" containerID="cri-o://e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753" gracePeriod=45
Oct 30 19:20:49.736139 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.736117    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:20:49.792287 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.792230    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-dshm\") pod \"8471ed21-b031-4f51-92f1-9ad6f069eae2\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") "
Oct 30 19:20:49.792287 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.792270    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8471ed21-b031-4f51-92f1-9ad6f069eae2-kube-api-access-wrhnl\" (UniqueName: \"kubernetes.io/projected/8471ed21-b031-4f51-92f1-9ad6f069eae2-kube-api-access-wrhnl\") pod \"8471ed21-b031-4f51-92f1-9ad6f069eae2\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") "
Oct 30 19:20:49.792421 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.792324    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-kserve-provision-location\") pod \"8471ed21-b031-4f51-92f1-9ad6f069eae2\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") "
Oct 30 19:20:49.792421 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.792369    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-tmp-dir\") pod \"8471ed21-b031-4f51-92f1-9ad6f069eae2\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") "
Oct 30 19:20:49.792421 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.792398    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-model-cache\") pod \"8471ed21-b031-4f51-92f1-9ad6f069eae2\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") "
Oct 30 19:20:49.792564 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.792474    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8471ed21-b031-4f51-92f1-9ad6f069eae2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8471ed21-b031-4f51-92f1-9ad6f069eae2-tls-certs\") pod \"8471ed21-b031-4f51-92f1-9ad6f069eae2\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") "
Oct 30 19:20:49.792564 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.792501    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-home\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-home\") pod \"8471ed21-b031-4f51-92f1-9ad6f069eae2\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") "
Oct 30 19:20:49.792564 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.792548    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8471ed21-b031-4f51-92f1-9ad6f069eae2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8471ed21-b031-4f51-92f1-9ad6f069eae2-cabundle-cert\") pod \"8471ed21-b031-4f51-92f1-9ad6f069eae2\" (UID: \"8471ed21-b031-4f51-92f1-9ad6f069eae2\") "
Oct 30 19:20:49.793226 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.793170    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-model-cache" pod "8471ed21-b031-4f51-92f1-9ad6f069eae2" (UID: "8471ed21-b031-4f51-92f1-9ad6f069eae2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:49.793453 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.793429    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8471ed21-b031-4f51-92f1-9ad6f069eae2-cabundle-cert" pod "8471ed21-b031-4f51-92f1-9ad6f069eae2" (UID: "8471ed21-b031-4f51-92f1-9ad6f069eae2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:49.793695 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.793663    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-home" pod "8471ed21-b031-4f51-92f1-9ad6f069eae2" (UID: "8471ed21-b031-4f51-92f1-9ad6f069eae2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:49.794822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.794796    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-dshm" pod "8471ed21-b031-4f51-92f1-9ad6f069eae2" (UID: "8471ed21-b031-4f51-92f1-9ad6f069eae2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:49.794908 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.794824    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8471ed21-b031-4f51-92f1-9ad6f069eae2-kube-api-access-wrhnl" pod "8471ed21-b031-4f51-92f1-9ad6f069eae2" (UID: "8471ed21-b031-4f51-92f1-9ad6f069eae2"). InnerVolumeSpecName "kube-api-access-wrhnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:49.795013 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.794991    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8471ed21-b031-4f51-92f1-9ad6f069eae2-tls-certs" pod "8471ed21-b031-4f51-92f1-9ad6f069eae2" (UID: "8471ed21-b031-4f51-92f1-9ad6f069eae2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:49.811075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.811027    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-tmp-dir" pod "8471ed21-b031-4f51-92f1-9ad6f069eae2" (UID: "8471ed21-b031-4f51-92f1-9ad6f069eae2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:49.815807 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.815788    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-kserve-provision-location" pod "8471ed21-b031-4f51-92f1-9ad6f069eae2" (UID: "8471ed21-b031-4f51-92f1-9ad6f069eae2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:49.872133 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:20:49.872102    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:20:49.893231 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.893214    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8471ed21-b031-4f51-92f1-9ad6f069eae2-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.893231 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.893233    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.893356 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.893241    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8471ed21-b031-4f51-92f1-9ad6f069eae2-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.893356 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.893250    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.893356 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.893279    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrhnl\" (UniqueName: \"kubernetes.io/projected/8471ed21-b031-4f51-92f1-9ad6f069eae2-kube-api-access-wrhnl\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.893356 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.893289    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.893356 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.893298    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:49.893356 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:49.893306    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8471ed21-b031-4f51-92f1-9ad6f069eae2-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:50.341851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.341810    4853 generic.go:356] "Generic (PLEG): container finished" podID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerID="e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753" exitCode=0
Oct 30 19:20:50.342262 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.341902    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"
Oct 30 19:20:50.342262 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.341909    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" event={"ID":"8471ed21-b031-4f51-92f1-9ad6f069eae2","Type":"ContainerDied","Data":"e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753"}
Oct 30 19:20:50.342262 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.341968    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb" event={"ID":"8471ed21-b031-4f51-92f1-9ad6f069eae2","Type":"ContainerDied","Data":"f352753e308c835a9292ea3216cc1283d4af7de6092e5de38f1a6a67c2aabc9f"}
Oct 30 19:20:50.342262 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.341994    4853 scope.go:122] "RemoveContainer" containerID="e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753"
Oct 30 19:20:50.351660 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.351634    4853 scope.go:122] "RemoveContainer" containerID="a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca"
Oct 30 19:20:50.359761 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.359737    4853 scope.go:122] "RemoveContainer" containerID="e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753"
Oct 30 19:20:50.360111 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:20:50.360087    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753\": container with ID starting with e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753 not found: ID does not exist" containerID="e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753"
Oct 30 19:20:50.360193 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.360125    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753"} err="failed to get container status \"e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753\": rpc error: code = NotFound desc = could not find container \"e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753\": container with ID starting with e9084fa95fa43d802d897ae4d4e3b763e67518758180dc13131b2e2d16a15753 not found: ID does not exist"
Oct 30 19:20:50.360193 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.360151    4853 scope.go:122] "RemoveContainer" containerID="a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca"
Oct 30 19:20:50.360452 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:20:50.360433    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca\": container with ID starting with a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca not found: ID does not exist" containerID="a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca"
Oct 30 19:20:50.360513 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.360462    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca"} err="failed to get container status \"a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca\": rpc error: code = NotFound desc = could not find container \"a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca\": container with ID starting with a60199042934a51b0da77bab730feba06b73aa7fcf8b8a175f07c39fa6e20cca not found: ID does not exist"
Oct 30 19:20:50.501704 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.501672    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"]
Oct 30 19:20:50.548319 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:50.548294    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-d97cf856-zfdxb"]
Oct 30 19:20:51.021462 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:51.021435    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:51.940075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:51.940038    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" path="/var/lib/kubelet/pods/8471ed21-b031-4f51-92f1-9ad6f069eae2/volumes"
Oct 30 19:20:58.297184 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:58.297144    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:58.600435 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:20:58.600406    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:02.973630 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:02.973587    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:05.771848 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:05.771816    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:12.228693 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:12.228656    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:12.439818 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:12.439789    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:16.026595 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:16.026563    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:16.490838 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:16.489426    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:19.884488 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:21:19.884447    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:21:22.443165 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:22.443129    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:22.679820 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:22.679790    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:23.513710 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:23.513681    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:23.897007 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:23.896975    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:24.061033 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.061003    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"]
Oct 30 19:21:24.062547 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.062508    4853 cpu_manager.go:414] "containerMap: removing container" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="storage-initializer"
Oct 30 19:21:24.062713 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.062700    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="storage-initializer"
Oct 30 19:21:24.062816 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.062807    4853 cpu_manager.go:414] "containerMap: removing container" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main"
Oct 30 19:21:24.062887 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.062877    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main"
Oct 30 19:21:24.063143 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.063131    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="8471ed21-b031-4f51-92f1-9ad6f069eae2" containerName="main"
Oct 30 19:21:24.067975 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.067952    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.118553 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.118512    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Oct 30 19:21:24.142801 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.142772    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.142932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.142807    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.142932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.142826    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-podinfo\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.142932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.142858    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-credential-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.143055 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.142924    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-data\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.143055 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.143043    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-certs\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.143129 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.143114    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-token\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.143179 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.143141    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-envoy\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.143259 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.143237    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfjw\" (UniqueName: \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-kube-api-access-bnfjw\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.144312 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.144296    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:21:24.144432 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.144339    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Oct 30 19:21:24.164772 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.164709    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-xblb5\""
Oct 30 19:21:24.167713 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.167692    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"]
Oct 30 19:21:24.227857 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.227805    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:24.243851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.243817    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-token\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.243973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.243857    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-envoy\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.243973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.243885    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfjw\" (UniqueName: \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-kube-api-access-bnfjw\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.243973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.243923    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.243973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.243938    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.243973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.243955    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-podinfo\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.244239 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.244002    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-credential-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.244239 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.244025    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-data\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.244239 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.244053    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-certs\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.244471 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.244446    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.244683 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.244657    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-data\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.244772 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.244713    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-certs\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.244833 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.244781    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.244927 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.244906    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-credential-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.246771 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.246747    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-envoy\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.246893 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.246842    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-podinfo\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.306096 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.306075    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-token\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.306220 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.306201    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfjw\" (UniqueName: \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-kube-api-access-bnfjw\") pod \"router-gateway-1-openshift-default-56d96d6b69-nn5vb\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.381133 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.381110    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:24.488085 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.488052    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:24.802904 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.802833    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"]
Oct 30 19:21:24.809519 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:21:24.809495    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeeabb93_1754_4955_a1c8_90b3b9d3f18d.slice/crio-319849f09e120aa51b49aab3fdb493ea0a6f7dc3ad3032b68fba92d8f55157fd WatchSource:0}: Error finding container 319849f09e120aa51b49aab3fdb493ea0a6f7dc3ad3032b68fba92d8f55157fd: Status 404 returned error can't find the container with id 319849f09e120aa51b49aab3fdb493ea0a6f7dc3ad3032b68fba92d8f55157fd
Oct 30 19:21:24.811491 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.811457    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:21:24.811570 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.811559    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:21:24.811615 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:24.811594    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:21:25.518033 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:25.517997    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" event={"ID":"deeabb93-1754-4955-a1c8-90b3b9d3f18d","Type":"ContainerStarted","Data":"8cecfb377be73982a02fedb0cd1c33638361e71963daabd21fb142aa1d1e3198"}
Oct 30 19:21:25.518216 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:25.518052    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" event={"ID":"deeabb93-1754-4955-a1c8-90b3b9d3f18d","Type":"ContainerStarted","Data":"319849f09e120aa51b49aab3fdb493ea0a6f7dc3ad3032b68fba92d8f55157fd"}
Oct 30 19:21:25.679788 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:25.679743    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" podStartSLOduration=1.679727988 podStartE2EDuration="1.679727988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:25.677949845 +0000 UTC m=+2591.848306692" watchObservedRunningTime="2026-10-30 19:21:25.679727988 +0000 UTC m=+2591.850084901"
Oct 30 19:21:26.381308 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:26.381271    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:26.386163 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:26.386141    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:26.522626 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:26.522603    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:26.523562 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:26.523508    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:27.449855 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:27.449824    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:28.609779 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:28.609743    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:30.637920 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.637885    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"]
Oct 30 19:21:30.641680 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.641660    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.693006 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.692890    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"]
Oct 30 19:21:30.698500 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.698476    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-dshm\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.698628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.698513    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kserve-provision-location\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.698628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.698552    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-home\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.698709 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.698673    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tls-certs\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.698789 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.698771    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzpwl\" (UniqueName: \"kubernetes.io/projected/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kube-api-access-nzpwl\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.698851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.698817    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tmp-dir\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.698928 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.698913    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-model-cache\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.698978 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.698964    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/382e0ffc-7701-42f2-becc-fc58b2f84c6d-cabundle-cert\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.737805 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.737771    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:21:30.737805 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.737771    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Oct 30 19:21:30.737980 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.737817    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-g8jws\""
Oct 30 19:21:30.737980 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.737881    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Oct 30 19:21:30.748387 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.748363    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:30.799793 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.799761    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzpwl\" (UniqueName: \"kubernetes.io/projected/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kube-api-access-nzpwl\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.799918 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.799803    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tmp-dir\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.799918 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.799835    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-model-cache\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.799918 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.799864    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/382e0ffc-7701-42f2-becc-fc58b2f84c6d-cabundle-cert\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.799918 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.799882    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-dshm\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.799918 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.799903    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kserve-provision-location\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.800187 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.799920    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-home\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.800187 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.799949    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tls-certs\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.800293 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.800239    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tmp-dir\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.800390 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.800365    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kserve-provision-location\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.800454 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.800376    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-home\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.800454 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.800435    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-model-cache\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.800663 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.800645    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/382e0ffc-7701-42f2-becc-fc58b2f84c6d-cabundle-cert\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.802601 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.802580    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-dshm\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.802953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.802934    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tls-certs\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.855548 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.855505    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzpwl\" (UniqueName: \"kubernetes.io/projected/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kube-api-access-nzpwl\") pod \"router-with-refs-test-kserve-cdc6d978d-lqp2n\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:30.959357 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:30.959280    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:31.243345 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:31.243264    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:31.437894 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:31.437869    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:31.443869 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:31.443846    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"]
Oct 30 19:21:31.449424 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:21:31.449394    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382e0ffc_7701_42f2_becc_fc58b2f84c6d.slice/crio-ea66f620fefb9a68eb12fc1b6cb9416e9d73799553f2507d0840f1b1af352d4b WatchSource:0}: Error finding container ea66f620fefb9a68eb12fc1b6cb9416e9d73799553f2507d0840f1b1af352d4b: Status 404 returned error can't find the container with id ea66f620fefb9a68eb12fc1b6cb9416e9d73799553f2507d0840f1b1af352d4b
Oct 30 19:21:31.553338 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:31.553308    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" event={"ID":"382e0ffc-7701-42f2-becc-fc58b2f84c6d","Type":"ContainerStarted","Data":"4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349"}
Oct 30 19:21:31.553499 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:31.553358    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" event={"ID":"382e0ffc-7701-42f2-becc-fc58b2f84c6d","Type":"ContainerStarted","Data":"ea66f620fefb9a68eb12fc1b6cb9416e9d73799553f2507d0840f1b1af352d4b"}
Oct 30 19:21:32.245427 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:32.245379    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:34.591757 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:34.591724    4853 generic.go:356] "Generic (PLEG): container finished" podID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerID="4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349" exitCode=0
Oct 30 19:21:34.592167 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:34.591791    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" event={"ID":"382e0ffc-7701-42f2-becc-fc58b2f84c6d","Type":"ContainerDied","Data":"4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349"}
Oct 30 19:21:35.600371 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:35.600336    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" event={"ID":"382e0ffc-7701-42f2-becc-fc58b2f84c6d","Type":"ContainerStarted","Data":"bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778"}
Oct 30 19:21:35.823443 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:35.823357    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podStartSLOduration=3.303877672 podStartE2EDuration="5.823338801s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.519461129s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:35.822047691 +0000 UTC m=+2601.992404538" watchObservedRunningTime="2026-10-30 19:21:35.823338801 +0000 UTC m=+2601.993695650"
Oct 30 19:21:35.960752 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:35.960614    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:39.139799 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:39.139751    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:39.340344 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:39.340300    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:40.959627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:40.959548    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:21:40.961404 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:40.961371    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:21:41.524841 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:41.524798    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 19:21:41.630687 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:41.630651    4853 generic.go:356] "Generic (PLEG): container finished" podID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerID="8cecfb377be73982a02fedb0cd1c33638361e71963daabd21fb142aa1d1e3198" exitCode=0
Oct 30 19:21:41.630842 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:41.630723    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" event={"ID":"deeabb93-1754-4955-a1c8-90b3b9d3f18d","Type":"ContainerDied","Data":"8cecfb377be73982a02fedb0cd1c33638361e71963daabd21fb142aa1d1e3198"}
Oct 30 19:21:41.631073 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:41.631058    4853 scope.go:122] "RemoveContainer" containerID="8cecfb377be73982a02fedb0cd1c33638361e71963daabd21fb142aa1d1e3198"
Oct 30 19:21:41.632131 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:41.632099    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:21:41.632212 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:41.632192    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:21:41.632261 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:41.632246    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:21:41.925848 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:41.925813    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:42.656851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:42.656813    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" event={"ID":"deeabb93-1754-4955-a1c8-90b3b9d3f18d","Type":"ContainerStarted","Data":"467b83b203f85d99494b6bd31ce31d990ce3e97ca8d03ea1a8cf97145fd01b9a"}
Oct 30 19:21:43.382012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:43.381976    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:43.382267 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:43.382237    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 19:21:44.382381 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:44.382338    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 19:21:45.381970 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:45.381923    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 19:21:46.382089 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:46.382045    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 19:21:47.382445 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:47.382393    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 19:21:48.385236 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:48.385202    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:48.385758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:48.385473    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:48.386132 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:48.386115    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:21:49.872794 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:21:49.872755    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:21:50.960007 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:21:50.959962    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:22:00.960329 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:22:00.960287    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:22:03.299311 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:22:03.299270    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:10.959919 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:22:10.959869    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:22:19.872471 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:22:19.872429    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:22:20.959848 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:22:20.959802    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:22:30.959949 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:22:30.959906    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:22:40.960635 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:22:40.960584    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:22:49.872605 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:22:49.872563    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:22:50.960619 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:22:50.960571    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:23:00.960179 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:00.960131    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:23:03.692687 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:03.692651    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:10.960656 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:10.960610    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:23:14.498317 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:14.498269    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:14.709543 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:14.709499    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:14.911067 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:14.911026    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:19.879304 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:23:19.879267    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:23:20.970222 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:20.970195    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:23:20.978287 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:20.978264    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:23:24.803995 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:24.803959    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:26.164247 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:26.164217    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:29.829972 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:29.829944    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"]
Oct 30 19:23:29.900602 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:29.900565    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:30.110487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:30.110388    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:37.141707 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:37.141674    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:44.853447 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:44.853406    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" containerID="cri-o://bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778" gracePeriod=45
Oct 30 19:23:44.971222 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:44.971180    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 19:23:45.494395 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.494371    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:23:45.550133 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.550110    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tls-certs\" (UniqueName: \"kubernetes.io/secret/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tls-certs\") pod \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") "
Oct 30 19:23:45.550252 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.550188    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-dshm\") pod \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") "
Oct 30 19:23:45.550252 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.550204    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-home\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-home\") pod \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") "
Oct 30 19:23:45.550252 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.550233    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kube-api-access-nzpwl\" (UniqueName: \"kubernetes.io/projected/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kube-api-access-nzpwl\") pod \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") "
Oct 30 19:23:45.550420 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.550287    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tmp-dir\") pod \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") "
Oct 30 19:23:45.550420 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.550320    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382e0ffc-7701-42f2-becc-fc58b2f84c6d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/382e0ffc-7701-42f2-becc-fc58b2f84c6d-cabundle-cert\") pod \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") "
Oct 30 19:23:45.550420 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.550366    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kserve-provision-location\") pod \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") "
Oct 30 19:23:45.550420 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.550393    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-model-cache\") pod \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\" (UID: \"382e0ffc-7701-42f2-becc-fc58b2f84c6d\") "
Oct 30 19:23:45.551051 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.550847    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-model-cache" pod "382e0ffc-7701-42f2-becc-fc58b2f84c6d" (UID: "382e0ffc-7701-42f2-becc-fc58b2f84c6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:45.551051 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.551006    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-home" pod "382e0ffc-7701-42f2-becc-fc58b2f84c6d" (UID: "382e0ffc-7701-42f2-becc-fc58b2f84c6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:45.551218 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.551199    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382e0ffc-7701-42f2-becc-fc58b2f84c6d-cabundle-cert" pod "382e0ffc-7701-42f2-becc-fc58b2f84c6d" (UID: "382e0ffc-7701-42f2-becc-fc58b2f84c6d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:45.552672 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.552649    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-dshm" pod "382e0ffc-7701-42f2-becc-fc58b2f84c6d" (UID: "382e0ffc-7701-42f2-becc-fc58b2f84c6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:45.552791 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.552671    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tls-certs" pod "382e0ffc-7701-42f2-becc-fc58b2f84c6d" (UID: "382e0ffc-7701-42f2-becc-fc58b2f84c6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:45.552852 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.552798    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kube-api-access-nzpwl" pod "382e0ffc-7701-42f2-becc-fc58b2f84c6d" (UID: "382e0ffc-7701-42f2-becc-fc58b2f84c6d"). InnerVolumeSpecName "kube-api-access-nzpwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:45.563104 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.563077    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tmp-dir" pod "382e0ffc-7701-42f2-becc-fc58b2f84c6d" (UID: "382e0ffc-7701-42f2-becc-fc58b2f84c6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:45.573618 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.573594    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kserve-provision-location" pod "382e0ffc-7701-42f2-becc-fc58b2f84c6d" (UID: "382e0ffc-7701-42f2-becc-fc58b2f84c6d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:45.651829 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.651767    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.651829 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.651793    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.651829 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.651808    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.651829 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.651820    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.652021 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.651835    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.652021 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.651847    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzpwl\" (UniqueName: \"kubernetes.io/projected/382e0ffc-7701-42f2-becc-fc58b2f84c6d-kube-api-access-nzpwl\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.652021 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.651859    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382e0ffc-7701-42f2-becc-fc58b2f84c6d-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:45.652021 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:45.651871    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/382e0ffc-7701-42f2-becc-fc58b2f84c6d-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:46.296827 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.296742    4853 generic.go:356] "Generic (PLEG): container finished" podID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerID="bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778" exitCode=0
Oct 30 19:23:46.296827 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.296813    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"
Oct 30 19:23:46.297331 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.296829    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" event={"ID":"382e0ffc-7701-42f2-becc-fc58b2f84c6d","Type":"ContainerDied","Data":"bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778"}
Oct 30 19:23:46.297331 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.296887    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n" event={"ID":"382e0ffc-7701-42f2-becc-fc58b2f84c6d","Type":"ContainerDied","Data":"ea66f620fefb9a68eb12fc1b6cb9416e9d73799553f2507d0840f1b1af352d4b"}
Oct 30 19:23:46.297331 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.296912    4853 scope.go:122] "RemoveContainer" containerID="bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778"
Oct 30 19:23:46.306438 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.306410    4853 scope.go:122] "RemoveContainer" containerID="4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349"
Oct 30 19:23:46.315019 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.314996    4853 scope.go:122] "RemoveContainer" containerID="bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778"
Oct 30 19:23:46.315341 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:23:46.315314    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778\": container with ID starting with bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778 not found: ID does not exist" containerID="bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778"
Oct 30 19:23:46.315392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.315350    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778"} err="failed to get container status \"bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778\": rpc error: code = NotFound desc = could not find container \"bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778\": container with ID starting with bceeb9f10e75c39c27b1cb2a67751e59c233183ce8ce629943fbc29adec51778 not found: ID does not exist"
Oct 30 19:23:46.315392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.315370    4853 scope.go:122] "RemoveContainer" containerID="4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349"
Oct 30 19:23:46.315673 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:23:46.315653    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349\": container with ID starting with 4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349 not found: ID does not exist" containerID="4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349"
Oct 30 19:23:46.315732 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.315680    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349"} err="failed to get container status \"4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349\": rpc error: code = NotFound desc = could not find container \"4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349\": container with ID starting with 4762afe305b7c6fc8ecebbb8f085ee56c27621bcd1fcf9ee871a3755fe4bb349 not found: ID does not exist"
Oct 30 19:23:46.459608 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.459583    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"]
Oct 30 19:23:46.506637 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:46.506611    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-cdc6d978d-lqp2n"]
Oct 30 19:23:47.938821 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:23:47.938792    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" path="/var/lib/kubelet/pods/382e0ffc-7701-42f2-becc-fc58b2f84c6d/volumes"
Oct 30 19:23:49.875650 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:23:49.875613    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:24:04.063192 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:04.063144    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:19.433602 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:19.433567    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:19.683061 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:19.683026    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:19.873438 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:24:19.873401    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:24:24.284808 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:24.284775    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"]
Oct 30 19:24:24.285238 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:24.284959    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy" containerID="cri-o://467b83b203f85d99494b6bd31ce31d990ce3e97ca8d03ea1a8cf97145fd01b9a" gracePeriod=30
Oct 30 19:24:24.415233 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:24.415204    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:26.790232 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.790200    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"]
Oct 30 19:24:26.790628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.790498    4853 cpu_manager.go:414] "containerMap: removing container" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main"
Oct 30 19:24:26.790628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.790508    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main"
Oct 30 19:24:26.790628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.790524    4853 cpu_manager.go:414] "containerMap: removing container" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="storage-initializer"
Oct 30 19:24:26.790628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.790543    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="storage-initializer"
Oct 30 19:24:26.790628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.790596    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="382e0ffc-7701-42f2-becc-fc58b2f84c6d" containerName="main"
Oct 30 19:24:26.795018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.794999    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:26.892044 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.892022    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Oct 30 19:24:26.892426 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.892404    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Oct 30 19:24:26.892633 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.892403    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Oct 30 19:24:26.892723 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.892431    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Oct 30 19:24:26.892723 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.892444    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-sl9c2\""
Oct 30 19:24:26.892723 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.892444    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:24:26.892723 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.892484    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-fzhfh\""
Oct 30 19:24:26.894791 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.894770    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"]
Oct 30 19:24:26.960083 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.960061    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:26.960211 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.960100    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:26.960211 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.960123    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:26.960211 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.960142    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:26.960315 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.960256    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:26.960315 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.960305    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql8cw\" (UniqueName: \"kubernetes.io/projected/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kube-api-access-ql8cw\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:26.960431 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.960411    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:26.960464 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:26.960446    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.049208 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.049135    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:27.060852 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.060830    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.060985 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.060859    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.060985 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.060884    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.060985 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.060910    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.060985 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.060941    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.060985 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.060962    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.060985 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.060979    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.061297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.061021    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql8cw\" (UniqueName: \"kubernetes.io/projected/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kube-api-access-ql8cw\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.061297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.061256    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.061405 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.061313    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.061566 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.061516    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.061690 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.061598    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.061895 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.061752    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.063345 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.063324    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.063641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.063624    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.117573 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.117547    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql8cw\" (UniqueName: \"kubernetes.io/projected/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kube-api-access-ql8cw\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.411390 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.411358    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:27.528943 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.528904    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:27.894240 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:27.894206    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"]
Oct 30 19:24:27.952071 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:24:27.952046    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b9e4e4_bbe4_485d_a180_a65b2bbd834b.slice/crio-809e88c2629df6bf4dad6da434b1e8d960534e31f9110a0d312f73d0ed807a76 WatchSource:0}: Error finding container 809e88c2629df6bf4dad6da434b1e8d960534e31f9110a0d312f73d0ed807a76: Status 404 returned error can't find the container with id 809e88c2629df6bf4dad6da434b1e8d960534e31f9110a0d312f73d0ed807a76
Oct 30 19:24:28.533266 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:28.533221    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" event={"ID":"71b9e4e4-bbe4-485d-a180-a65b2bbd834b","Type":"ContainerStarted","Data":"ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b"}
Oct 30 19:24:28.533479 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:28.533282    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" event={"ID":"71b9e4e4-bbe4-485d-a180-a65b2bbd834b","Type":"ContainerStarted","Data":"809e88c2629df6bf4dad6da434b1e8d960534e31f9110a0d312f73d0ed807a76"}
Oct 30 19:24:28.533479 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:28.533349    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:29.540269 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.540244    4853 generic.go:356] "Generic (PLEG): container finished" podID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerID="467b83b203f85d99494b6bd31ce31d990ce3e97ca8d03ea1a8cf97145fd01b9a" exitCode=0
Oct 30 19:24:29.540685 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.540367    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" event={"ID":"deeabb93-1754-4955-a1c8-90b3b9d3f18d","Type":"ContainerDied","Data":"467b83b203f85d99494b6bd31ce31d990ce3e97ca8d03ea1a8cf97145fd01b9a"}
Oct 30 19:24:29.540685 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.540422    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb" event={"ID":"deeabb93-1754-4955-a1c8-90b3b9d3f18d","Type":"ContainerDied","Data":"319849f09e120aa51b49aab3fdb493ea0a6f7dc3ad3032b68fba92d8f55157fd"}
Oct 30 19:24:29.540685 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.540446    4853 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="319849f09e120aa51b49aab3fdb493ea0a6f7dc3ad3032b68fba92d8f55157fd"
Oct 30 19:24:29.540685 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.540464    4853 scope.go:122] "RemoveContainer" containerID="8cecfb377be73982a02fedb0cd1c33638361e71963daabd21fb142aa1d1e3198"
Oct 30 19:24:29.543119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.543092    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" event={"ID":"71b9e4e4-bbe4-485d-a180-a65b2bbd834b","Type":"ContainerStarted","Data":"967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11"}
Oct 30 19:24:29.547252 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.547186    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:24:29.586760 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.586731    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-certs\") pod \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") "
Oct 30 19:24:29.586884 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.586813    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-token\" (UniqueName: \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-token\") pod \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") "
Oct 30 19:24:29.586884 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.586861    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istiod-ca-cert\") pod \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") "
Oct 30 19:24:29.587001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.586902    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-data\") pod \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") "
Oct 30 19:24:29.587001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.586936    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-credential-socket\") pod \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") "
Oct 30 19:24:29.587001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.586963    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-certs" pod "deeabb93-1754-4955-a1c8-90b3b9d3f18d" (UID: "deeabb93-1754-4955-a1c8-90b3b9d3f18d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:29.587001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.586984    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-socket\") pod \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") "
Oct 30 19:24:29.587217 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.587013    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-kube-api-access-bnfjw\" (UniqueName: \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-kube-api-access-bnfjw\") pod \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") "
Oct 30 19:24:29.587266 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.587251    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istiod-ca-cert" pod "deeabb93-1754-4955-a1c8-90b3b9d3f18d" (UID: "deeabb93-1754-4955-a1c8-90b3b9d3f18d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:29.587348 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.587323    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:29.587348 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.587346    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:29.587632 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.587590    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-socket" pod "deeabb93-1754-4955-a1c8-90b3b9d3f18d" (UID: "deeabb93-1754-4955-a1c8-90b3b9d3f18d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:29.587695 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.587660    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-credential-socket" pod "deeabb93-1754-4955-a1c8-90b3b9d3f18d" (UID: "deeabb93-1754-4955-a1c8-90b3b9d3f18d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:29.587987 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.587967    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-data" pod "deeabb93-1754-4955-a1c8-90b3b9d3f18d" (UID: "deeabb93-1754-4955-a1c8-90b3b9d3f18d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:29.589645 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.589619    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-kube-api-access-bnfjw" pod "deeabb93-1754-4955-a1c8-90b3b9d3f18d" (UID: "deeabb93-1754-4955-a1c8-90b3b9d3f18d"). InnerVolumeSpecName "kube-api-access-bnfjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:29.589742 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.589722    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-token" pod "deeabb93-1754-4955-a1c8-90b3b9d3f18d" (UID: "deeabb93-1754-4955-a1c8-90b3b9d3f18d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:29.687616 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.687587    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-podinfo\") pod \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") "
Oct 30 19:24:29.687788 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.687625    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-envoy\") pod \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\" (UID: \"deeabb93-1754-4955-a1c8-90b3b9d3f18d\") "
Oct 30 19:24:29.687900 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.687880    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:29.687944 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.687907    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:29.687982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.687941    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:29.687982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.687958    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:29.687982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.687974    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnfjw\" (UniqueName: \"kubernetes.io/projected/deeabb93-1754-4955-a1c8-90b3b9d3f18d-kube-api-access-bnfjw\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:29.689805 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.689779    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-envoy" pod "deeabb93-1754-4955-a1c8-90b3b9d3f18d" (UID: "deeabb93-1754-4955-a1c8-90b3b9d3f18d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:29.689805 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.689795    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-podinfo" pod "deeabb93-1754-4955-a1c8-90b3b9d3f18d" (UID: "deeabb93-1754-4955-a1c8-90b3b9d3f18d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:24:29.788330 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.788299    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:29.788330 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:29.788323    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deeabb93-1754-4955-a1c8-90b3b9d3f18d-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:30.550903 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:30.550871    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"
Oct 30 19:24:30.770991 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:30.770958    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"]
Oct 30 19:24:30.778195 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:30.778161    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-nn5vb"]
Oct 30 19:24:31.557186 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:31.557153    4853 generic.go:356] "Generic (PLEG): container finished" podID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerID="967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11" exitCode=0
Oct 30 19:24:31.557658 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:31.557194    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" event={"ID":"71b9e4e4-bbe4-485d-a180-a65b2bbd834b","Type":"ContainerDied","Data":"967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11"}
Oct 30 19:24:31.940154 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:31.940117    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" path="/var/lib/kubelet/pods/deeabb93-1754-4955-a1c8-90b3b9d3f18d/volumes"
Oct 30 19:24:32.564736 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:32.564698    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" event={"ID":"71b9e4e4-bbe4-485d-a180-a65b2bbd834b","Type":"ContainerStarted","Data":"38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2"}
Oct 30 19:24:32.777617 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:32.777558    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podStartSLOduration=4.498035241 podStartE2EDuration="6.777541885s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.279506644s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:32.729662513 +0000 UTC m=+2778.900019362" watchObservedRunningTime="2026-10-30 19:24:32.777541885 +0000 UTC m=+2778.947898734"
Oct 30 19:24:33.144687 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:33.144651    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"]
Oct 30 19:24:33.144972 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:33.144936    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerName="istio-proxy" containerID="cri-o://7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a" gracePeriod=30
Oct 30 19:24:33.147166 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:33.147135    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"]
Oct 30 19:24:33.147364 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:33.147337    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" podUID="f60c4763-4f32-48c7-8063-e977053d11d4" containerName="istio-proxy" containerID="cri-o://4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6" gracePeriod=30
Oct 30 19:24:33.149285 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:33.149247    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"]
Oct 30 19:24:33.149509 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:33.149460    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerName="istio-proxy" containerID="cri-o://20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925" gracePeriod=30
Oct 30 19:24:33.151047 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:33.151027    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"]
Oct 30 19:24:33.151209 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:33.151181    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" podUID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" containerName="istio-proxy" containerID="cri-o://da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00" gracePeriod=30
Oct 30 19:24:37.145685 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:37.145637    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:37.411504 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:37.411468    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:37.411709 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:37.411565    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:37.412932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:37.412893    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Oct 30 19:24:37.954549 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:37.954493    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:24:37.954723 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:37.954588    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" podUID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:24:38.001127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.001099    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-phjn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:24:38.001271 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.001144    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" podUID="f60c4763-4f32-48c7-8063-e977053d11d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:24:38.079884 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.079849    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-zskzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:24:38.080018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.079906    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:24:38.128883 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.128838    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:24:38.128999 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.128900    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:24:38.429788 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.429754    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:24:38.454075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.453247    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-kube-api-access-nctls\" (UniqueName: \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-kube-api-access-nctls\") pod \"0ee75a04-5218-4428-b5f2-eb0403792f33\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") "
Oct 30 19:24:38.454075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.453310    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-socket\") pod \"0ee75a04-5218-4428-b5f2-eb0403792f33\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") "
Oct 30 19:24:38.454075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.453340    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee75a04-5218-4428-b5f2-eb0403792f33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee75a04-5218-4428-b5f2-eb0403792f33-istiod-ca-cert\") pod \"0ee75a04-5218-4428-b5f2-eb0403792f33\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") "
Oct 30 19:24:38.454075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.453381    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-data\") pod \"0ee75a04-5218-4428-b5f2-eb0403792f33\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") "
Oct 30 19:24:38.454075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.453413    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-envoy\") pod \"0ee75a04-5218-4428-b5f2-eb0403792f33\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") "
Oct 30 19:24:38.454075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.453469    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-podinfo\") pod \"0ee75a04-5218-4428-b5f2-eb0403792f33\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") "
Oct 30 19:24:38.454075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.453494    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-certs\") pod \"0ee75a04-5218-4428-b5f2-eb0403792f33\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") "
Oct 30 19:24:38.454075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.453578    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-credential-socket\") pod \"0ee75a04-5218-4428-b5f2-eb0403792f33\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") "
Oct 30 19:24:38.454075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.453606    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-token\" (UniqueName: \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-token\") pod \"0ee75a04-5218-4428-b5f2-eb0403792f33\" (UID: \"0ee75a04-5218-4428-b5f2-eb0403792f33\") "
Oct 30 19:24:38.454725 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.454661    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee75a04-5218-4428-b5f2-eb0403792f33-istiod-ca-cert" pod "0ee75a04-5218-4428-b5f2-eb0403792f33" (UID: "0ee75a04-5218-4428-b5f2-eb0403792f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:38.454803 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.454711    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-data" pod "0ee75a04-5218-4428-b5f2-eb0403792f33" (UID: "0ee75a04-5218-4428-b5f2-eb0403792f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.454803 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.454784    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-socket" pod "0ee75a04-5218-4428-b5f2-eb0403792f33" (UID: "0ee75a04-5218-4428-b5f2-eb0403792f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.454910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.454633    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-certs" pod "0ee75a04-5218-4428-b5f2-eb0403792f33" (UID: "0ee75a04-5218-4428-b5f2-eb0403792f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.454910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.454835    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-credential-socket" pod "0ee75a04-5218-4428-b5f2-eb0403792f33" (UID: "0ee75a04-5218-4428-b5f2-eb0403792f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.458316 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.458291    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-envoy" pod "0ee75a04-5218-4428-b5f2-eb0403792f33" (UID: "0ee75a04-5218-4428-b5f2-eb0403792f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.459318 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.459284    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-kube-api-access-nctls" pod "0ee75a04-5218-4428-b5f2-eb0403792f33" (UID: "0ee75a04-5218-4428-b5f2-eb0403792f33"). InnerVolumeSpecName "kube-api-access-nctls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:38.459432 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.459383    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-token" pod "0ee75a04-5218-4428-b5f2-eb0403792f33" (UID: "0ee75a04-5218-4428-b5f2-eb0403792f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:38.459432 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.459402    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-podinfo" pod "0ee75a04-5218-4428-b5f2-eb0403792f33" (UID: "0ee75a04-5218-4428-b5f2-eb0403792f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:24:38.476418 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.476399    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:24:38.486431 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.486413    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:24:38.489492 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.489475    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:24:38.554899 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.554826    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-certs\") pod \"f60c4763-4f32-48c7-8063-e977053d11d4\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") "
Oct 30 19:24:38.554899 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.554877    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-kube-api-access-lswht\" (UniqueName: \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-kube-api-access-lswht\") pod \"f60c4763-4f32-48c7-8063-e977053d11d4\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") "
Oct 30 19:24:38.555090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.554906    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-data\") pod \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") "
Oct 30 19:24:38.555090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.554946    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-envoy\") pod \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") "
Oct 30 19:24:38.555090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.554980    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-certs\") pod \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") "
Oct 30 19:24:38.555090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555012    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-credential-socket\") pod \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") "
Oct 30 19:24:38.555090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555043    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-socket\") pod \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") "
Oct 30 19:24:38.555090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555075    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-credential-socket\") pod \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") "
Oct 30 19:24:38.555090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555081    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-certs" pod "f60c4763-4f32-48c7-8063-e977053d11d4" (UID: "f60c4763-4f32-48c7-8063-e977053d11d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555101    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-kube-api-access-pt6lg\" (UniqueName: \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-kube-api-access-pt6lg\") pod \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") "
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555140    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-token\" (UniqueName: \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-token\") pod \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") "
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555189    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f60c4763-4f32-48c7-8063-e977053d11d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60c4763-4f32-48c7-8063-e977053d11d4-istio-podinfo\") pod \"f60c4763-4f32-48c7-8063-e977053d11d4\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") "
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555200    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-data" pod "ed7713d2-baf3-4ac9-b046-9c944e5f629e" (UID: "ed7713d2-baf3-4ac9-b046-9c944e5f629e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555217    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-istio-token\" (UniqueName: \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-istio-token\") pod \"f60c4763-4f32-48c7-8063-e977053d11d4\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") "
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555253    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-data\") pod \"f60c4763-4f32-48c7-8063-e977053d11d4\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") "
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555278    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-data\") pod \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") "
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555310    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-kube-api-access-2sgjl\" (UniqueName: \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-kube-api-access-2sgjl\") pod \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") "
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555345    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-credential-socket" pod "ed7713d2-baf3-4ac9-b046-9c944e5f629e" (UID: "ed7713d2-baf3-4ac9-b046-9c944e5f629e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555357    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-credential-socket\") pod \"f60c4763-4f32-48c7-8063-e977053d11d4\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") "
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555410    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-certs" pod "ed7713d2-baf3-4ac9-b046-9c944e5f629e" (UID: "ed7713d2-baf3-4ac9-b046-9c944e5f629e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.555487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555430    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-socket\") pod \"f60c4763-4f32-48c7-8063-e977053d11d4\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") "
Oct 30 19:24:38.556146 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.555795    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-socket" pod "f60c4763-4f32-48c7-8063-e977053d11d4" (UID: "f60c4763-4f32-48c7-8063-e977053d11d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.556146 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.556038    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-socket" pod "ed7713d2-baf3-4ac9-b046-9c944e5f629e" (UID: "ed7713d2-baf3-4ac9-b046-9c944e5f629e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.556442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.556400    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-data" pod "f60c4763-4f32-48c7-8063-e977053d11d4" (UID: "f60c4763-4f32-48c7-8063-e977053d11d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.558137 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.556464    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-certs\") pod \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") "
Oct 30 19:24:38.558511 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.558486    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-socket\") pod \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") "
Oct 30 19:24:38.558669 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.558556    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-token\" (UniqueName: \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-token\") pod \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") "
Oct 30 19:24:38.558669 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.558611    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f60c4763-4f32-48c7-8063-e977053d11d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60c4763-4f32-48c7-8063-e977053d11d4-istiod-ca-cert\") pod \"f60c4763-4f32-48c7-8063-e977053d11d4\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") "
Oct 30 19:24:38.558669 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.558658    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-envoy\") pod \"f60c4763-4f32-48c7-8063-e977053d11d4\" (UID: \"f60c4763-4f32-48c7-8063-e977053d11d4\") "
Oct 30 19:24:38.558843 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.558705    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-podinfo\") pod \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") "
Oct 30 19:24:38.558843 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.558754    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istiod-ca-cert\") pod \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\" (UID: \"ed7713d2-baf3-4ac9-b046-9c944e5f629e\") "
Oct 30 19:24:38.558843 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.558783    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-podinfo\") pod \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") "
Oct 30 19:24:38.559641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.559604    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-certs" pod "f075728f-b71d-4b4d-82ea-3d10ec3b80c7" (UID: "f075728f-b71d-4b4d-82ea-3d10ec3b80c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.559830 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.559801    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-socket" pod "f075728f-b71d-4b4d-82ea-3d10ec3b80c7" (UID: "f075728f-b71d-4b4d-82ea-3d10ec3b80c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.561056 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.560780    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60c4763-4f32-48c7-8063-e977053d11d4-istiod-ca-cert" pod "f60c4763-4f32-48c7-8063-e977053d11d4" (UID: "f60c4763-4f32-48c7-8063-e977053d11d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:38.561056 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.560871    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-credential-socket" pod "f075728f-b71d-4b4d-82ea-3d10ec3b80c7" (UID: "f075728f-b71d-4b4d-82ea-3d10ec3b80c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.561056 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.560931    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-data" pod "f075728f-b71d-4b4d-82ea-3d10ec3b80c7" (UID: "f075728f-b71d-4b4d-82ea-3d10ec3b80c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.561056 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.560973    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-credential-socket" pod "f60c4763-4f32-48c7-8063-e977053d11d4" (UID: "f60c4763-4f32-48c7-8063-e977053d11d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.561361 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.561086    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-kube-api-access-pt6lg" pod "f075728f-b71d-4b4d-82ea-3d10ec3b80c7" (UID: "f075728f-b71d-4b4d-82ea-3d10ec3b80c7"). InnerVolumeSpecName "kube-api-access-pt6lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:38.562723 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.562695    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istiod-ca-cert" pod "ed7713d2-baf3-4ac9-b046-9c944e5f629e" (UID: "ed7713d2-baf3-4ac9-b046-9c944e5f629e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:38.563245 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.563036    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-kube-api-access-lswht" pod "f60c4763-4f32-48c7-8063-e977053d11d4" (UID: "f60c4763-4f32-48c7-8063-e977053d11d4"). InnerVolumeSpecName "kube-api-access-lswht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:38.563245 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.563129    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-envoy" pod "f60c4763-4f32-48c7-8063-e977053d11d4" (UID: "f60c4763-4f32-48c7-8063-e977053d11d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.563245 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.563148    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-token" pod "ed7713d2-baf3-4ac9-b046-9c944e5f629e" (UID: "ed7713d2-baf3-4ac9-b046-9c944e5f629e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:38.563245 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.563199    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-kube-api-access-2sgjl" pod "ed7713d2-baf3-4ac9-b046-9c944e5f629e" (UID: "ed7713d2-baf3-4ac9-b046-9c944e5f629e"). InnerVolumeSpecName "kube-api-access-2sgjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:38.563518 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.563237    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-envoy" pod "f075728f-b71d-4b4d-82ea-3d10ec3b80c7" (UID: "f075728f-b71d-4b4d-82ea-3d10ec3b80c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.563518 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.563349    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-envoy\") pod \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") "
Oct 30 19:24:38.563518 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.563400    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istiod-ca-cert\") pod \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\" (UID: \"f075728f-b71d-4b4d-82ea-3d10ec3b80c7\") "
Oct 30 19:24:38.566627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.563861    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.566627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.563894    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.566627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.563909    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.566627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.564130    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istiod-ca-cert" pod "f075728f-b71d-4b4d-82ea-3d10ec3b80c7" (UID: "f075728f-b71d-4b4d-82ea-3d10ec3b80c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:38.566627 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.565016    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-podinfo" pod "f075728f-b71d-4b4d-82ea-3d10ec3b80c7" (UID: "f075728f-b71d-4b4d-82ea-3d10ec3b80c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:24:38.569465 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.563927    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sgjl\" (UniqueName: \"kubernetes.io/projected/ed7713d2-baf3-4ac9-b046-9c944e5f629e-kube-api-access-2sgjl\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569585 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569484    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569585 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569502    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569585 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569517    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569585 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569566    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569585 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569581    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569595    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569611    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60c4763-4f32-48c7-8063-e977053d11d4-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569630    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569644    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569658    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569662    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-envoy" pod "ed7713d2-baf3-4ac9-b046-9c944e5f629e" (UID: "ed7713d2-baf3-4ac9-b046-9c944e5f629e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569671    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569706    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569724    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60c4763-4f32-48c7-8063-e977053d11d4-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569746    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lswht\" (UniqueName: \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-kube-api-access-lswht\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569760    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569774    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nctls\" (UniqueName: \"kubernetes.io/projected/0ee75a04-5218-4428-b5f2-eb0403792f33-kube-api-access-nctls\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569788    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569805    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee75a04-5218-4428-b5f2-eb0403792f33-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569820    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569834    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.569867 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569872    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.570753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569886    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.570753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569900    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.570753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569913    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt6lg\" (UniqueName: \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-kube-api-access-pt6lg\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.570753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.569932    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee75a04-5218-4428-b5f2-eb0403792f33-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.570753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.570589    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f60c4763-4f32-48c7-8063-e977053d11d4-istio-podinfo" pod "f60c4763-4f32-48c7-8063-e977053d11d4" (UID: "f60c4763-4f32-48c7-8063-e977053d11d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:24:38.571370 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.571344    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-token" pod "f075728f-b71d-4b4d-82ea-3d10ec3b80c7" (UID: "f075728f-b71d-4b4d-82ea-3d10ec3b80c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:38.571882 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.571855    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-podinfo" pod "ed7713d2-baf3-4ac9-b046-9c944e5f629e" (UID: "ed7713d2-baf3-4ac9-b046-9c944e5f629e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:24:38.571969 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.571910    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-istio-token" pod "f60c4763-4f32-48c7-8063-e977053d11d4" (UID: "f60c4763-4f32-48c7-8063-e977053d11d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:38.597848 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.597821    4853 generic.go:356] "Generic (PLEG): container finished" podID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerID="7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a" exitCode=0
Oct 30 19:24:38.597938 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.597861    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" event={"ID":"0ee75a04-5218-4428-b5f2-eb0403792f33","Type":"ContainerDied","Data":"7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a"}
Oct 30 19:24:38.597938 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.597888    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"
Oct 30 19:24:38.598050 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.597952    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv" event={"ID":"0ee75a04-5218-4428-b5f2-eb0403792f33","Type":"ContainerDied","Data":"3dff2c767629ddf9ddc78d5efb241a9ed18c2e22eb5b9a7ee9e0e3966ae57c37"}
Oct 30 19:24:38.598050 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.597979    4853 scope.go:122] "RemoveContainer" containerID="7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a"
Oct 30 19:24:38.599474 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.599453    4853 generic.go:356] "Generic (PLEG): container finished" podID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" containerID="da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00" exitCode=0
Oct 30 19:24:38.599605 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.599551    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" event={"ID":"ed7713d2-baf3-4ac9-b046-9c944e5f629e","Type":"ContainerDied","Data":"da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00"}
Oct 30 19:24:38.599605 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.599591    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z" event={"ID":"ed7713d2-baf3-4ac9-b046-9c944e5f629e","Type":"ContainerDied","Data":"0778175c5caeaeccafe9a0cab92250acbdafcd4deb0f8c169523a456ea1d01fd"}
Oct 30 19:24:38.599880 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.599864    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"
Oct 30 19:24:38.606319 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.606295    4853 generic.go:356] "Generic (PLEG): container finished" podID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerID="20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925" exitCode=0
Oct 30 19:24:38.606430 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.606363    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" event={"ID":"f075728f-b71d-4b4d-82ea-3d10ec3b80c7","Type":"ContainerDied","Data":"20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925"}
Oct 30 19:24:38.606495 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.606432    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f" event={"ID":"f075728f-b71d-4b4d-82ea-3d10ec3b80c7","Type":"ContainerDied","Data":"daec5f9f5649409a3ae5c2ea52a822d5ec19e78eecb485c955032391df275986"}
Oct 30 19:24:38.606912 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.606892    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"
Oct 30 19:24:38.608202 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.608182    4853 generic.go:356] "Generic (PLEG): container finished" podID="f60c4763-4f32-48c7-8063-e977053d11d4" containerID="4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6" exitCode=0
Oct 30 19:24:38.608313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.608240    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" event={"ID":"f60c4763-4f32-48c7-8063-e977053d11d4","Type":"ContainerDied","Data":"4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6"}
Oct 30 19:24:38.608313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.608277    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7" event={"ID":"f60c4763-4f32-48c7-8063-e977053d11d4","Type":"ContainerDied","Data":"8cb5597456c92c5c0d6514ef9a3ba51f0decd7951bbe56558b6871a673657dc1"}
Oct 30 19:24:38.608313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.608282    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"
Oct 30 19:24:38.619593 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.618519    4853 scope.go:122] "RemoveContainer" containerID="7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a"
Oct 30 19:24:38.619593 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:24:38.618859    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a\": container with ID starting with 7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a not found: ID does not exist" containerID="7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a"
Oct 30 19:24:38.619593 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.618896    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a"} err="failed to get container status \"7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a\": rpc error: code = NotFound desc = could not find container \"7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a\": container with ID starting with 7b9639b57f374d4993d723b7801c83d89f6d9bc3a2d09180c846ede3e383055a not found: ID does not exist"
Oct 30 19:24:38.619593 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.618918    4853 scope.go:122] "RemoveContainer" containerID="da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00"
Oct 30 19:24:38.628591 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.628516    4853 scope.go:122] "RemoveContainer" containerID="da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00"
Oct 30 19:24:38.628849 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:24:38.628827    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00\": container with ID starting with da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00 not found: ID does not exist" containerID="da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00"
Oct 30 19:24:38.628926 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.628855    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00"} err="failed to get container status \"da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00\": rpc error: code = NotFound desc = could not find container \"da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00\": container with ID starting with da276d12b898dc154f0cb4d5059c1dc739046b0b7272d5a7a8ad2d2530e7eb00 not found: ID does not exist"
Oct 30 19:24:38.628926 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.628872    4853 scope.go:122] "RemoveContainer" containerID="20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925"
Oct 30 19:24:38.636256 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.636237    4853 scope.go:122] "RemoveContainer" containerID="20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925"
Oct 30 19:24:38.636555 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:24:38.636507    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925\": container with ID starting with 20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925 not found: ID does not exist" containerID="20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925"
Oct 30 19:24:38.636636 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.636549    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925"} err="failed to get container status \"20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925\": rpc error: code = NotFound desc = could not find container \"20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925\": container with ID starting with 20b5101fb99296624f72cc7ffd151837a4352e86647f5342969d45b50ec57925 not found: ID does not exist"
Oct 30 19:24:38.636636 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.636566    4853 scope.go:122] "RemoveContainer" containerID="4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6"
Oct 30 19:24:38.646338 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.646283    4853 scope.go:122] "RemoveContainer" containerID="4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6"
Oct 30 19:24:38.646743 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:24:38.646720    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6\": container with ID starting with 4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6 not found: ID does not exist" containerID="4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6"
Oct 30 19:24:38.646825 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.646752    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6"} err="failed to get container status \"4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6\": rpc error: code = NotFound desc = could not find container \"4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6\": container with ID starting with 4539223bd8e32cc3907ff007162d30578acdb389e860e82ad838915055e450e6 not found: ID does not exist"
Oct 30 19:24:38.671365 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.671337    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.671475 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.671368    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.671475 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.671383    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.671475 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.671396    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f075728f-b71d-4b4d-82ea-3d10ec3b80c7-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.671475 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.671405    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed7713d2-baf3-4ac9-b046-9c944e5f629e-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.671475 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.671414    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60c4763-4f32-48c7-8063-e977053d11d4-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.671475 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:38.671422    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60c4763-4f32-48c7-8063-e977053d11d4-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:39.038050 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.038016    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"]
Oct 30 19:24:39.040800 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.040776    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-phjn7"]
Oct 30 19:24:39.147786 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.147754    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"]
Oct 30 19:24:39.154876 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.154850    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-m7s9f"]
Oct 30 19:24:39.259986 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.259957    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"]
Oct 30 19:24:39.265313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.265290    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nxn8z"]
Oct 30 19:24:39.370424 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.370396    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"]
Oct 30 19:24:39.376298 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.376273    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-zskzv"]
Oct 30 19:24:39.940336 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.940293    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" path="/var/lib/kubelet/pods/0ee75a04-5218-4428-b5f2-eb0403792f33/volumes"
Oct 30 19:24:39.940883 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.940714    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" path="/var/lib/kubelet/pods/ed7713d2-baf3-4ac9-b046-9c944e5f629e/volumes"
Oct 30 19:24:39.941081 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.941066    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" path="/var/lib/kubelet/pods/f075728f-b71d-4b4d-82ea-3d10ec3b80c7/volumes"
Oct 30 19:24:39.941757 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:39.941737    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f60c4763-4f32-48c7-8063-e977053d11d4" path="/var/lib/kubelet/pods/f60c4763-4f32-48c7-8063-e977053d11d4/volumes"
Oct 30 19:24:45.692773 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:45.692735    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:47.412548 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:47.412491    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Oct 30 19:24:47.609713 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:47.609678    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:24:48.563017 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:48.562978    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:49.872703 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:24:49.872667    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:24:54.971518 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:54.971467    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:57.412038 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:24:57.411992    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Oct 30 19:25:04.357338 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:04.357296    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:07.411752 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:07.411713    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Oct 30 19:25:07.767268 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:07.767167    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:10.932871 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:10.932831    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:11.209365 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:11.209288    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:17.411902 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:17.411855    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Oct 30 19:25:19.872794 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:25:19.872748    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:25:21.070233 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:21.070192    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:27.412616 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:27.412570    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Oct 30 19:25:32.726819 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:32.726786    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:32.951944 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:32.951900    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:37.412620 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:37.412577    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Oct 30 19:25:47.411689 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:47.411647    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Oct 30 19:25:49.872665 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:25:49.872621    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:25:57.412103 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:25:57.412046    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Oct 30 19:26:04.686149 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:04.686107    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:07.412241 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:07.412188    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Oct 30 19:26:17.422509 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:17.422472    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:26:17.434211 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:17.434188    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:26:19.865680 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:19.865642    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:19.873065 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:26:19.873038    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:26:20.194399 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:20.194303    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:27.197108 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:27.197070    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:34.210426 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:34.210396    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:34.424225 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:34.424196    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:35.864357 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864325    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"]
Oct 30 19:26:35.864822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864713    4853 cpu_manager.go:414] "containerMap: removing container" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy"
Oct 30 19:26:35.864822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864725    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy"
Oct 30 19:26:35.864822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864735    4853 cpu_manager.go:414] "containerMap: removing container" podUID="f60c4763-4f32-48c7-8063-e977053d11d4" containerName="istio-proxy"
Oct 30 19:26:35.864822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864740    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f60c4763-4f32-48c7-8063-e977053d11d4" containerName="istio-proxy"
Oct 30 19:26:35.864822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864750    4853 cpu_manager.go:414] "containerMap: removing container" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerName="istio-proxy"
Oct 30 19:26:35.864822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864754    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerName="istio-proxy"
Oct 30 19:26:35.864822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864789    4853 cpu_manager.go:414] "containerMap: removing container" podUID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" containerName="istio-proxy"
Oct 30 19:26:35.864822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864794    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" containerName="istio-proxy"
Oct 30 19:26:35.864822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864805    4853 cpu_manager.go:414] "containerMap: removing container" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerName="istio-proxy"
Oct 30 19:26:35.864822 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864812    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerName="istio-proxy"
Oct 30 19:26:35.865138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864862    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy"
Oct 30 19:26:35.865138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864868    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="f60c4763-4f32-48c7-8063-e977053d11d4" containerName="istio-proxy"
Oct 30 19:26:35.865138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864877    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee75a04-5218-4428-b5f2-eb0403792f33" containerName="istio-proxy"
Oct 30 19:26:35.865138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864884    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7713d2-baf3-4ac9-b046-9c944e5f629e" containerName="istio-proxy"
Oct 30 19:26:35.865138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864890    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="f075728f-b71d-4b4d-82ea-3d10ec3b80c7" containerName="istio-proxy"
Oct 30 19:26:35.865138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864950    4853 cpu_manager.go:414] "containerMap: removing container" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy"
Oct 30 19:26:35.865138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.864955    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy"
Oct 30 19:26:35.865138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.865009    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="deeabb93-1754-4955-a1c8-90b3b9d3f18d" containerName="istio-proxy"
Oct 30 19:26:35.869597 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.869579    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:35.918485 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.918453    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"]
Oct 30 19:26:35.962498 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962458    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tls-certs\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:35.962498 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962497    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-cabundle-cert\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:35.962734 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962551    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:35.962734 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962592    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-model-cache\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:35.962734 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962644    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:26:35.962734 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962648    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Oct 30 19:26:35.962734 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962658    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Oct 30 19:26:35.962734 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962676    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Oct 30 19:26:35.962734 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962714    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-d5bm5\""
Oct 30 19:26:35.962734 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962726    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Oct 30 19:26:35.963036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962738    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-dshm\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:35.963036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962776    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-home\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:35.963036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962846    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txcrp\" (UniqueName: \"kubernetes.io/projected/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kube-api-access-txcrp\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:35.963036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.962871    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tmp-dir\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:35.973518 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:35.973481    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:36.063950 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.063919    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-cabundle-cert\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.064074 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.063959    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.064074 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.063984    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-model-cache\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.064074 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.064020    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-dshm\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.064074 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.064040    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-home\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.064074 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.064056    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txcrp\" (UniqueName: \"kubernetes.io/projected/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kube-api-access-txcrp\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.064311 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.064080    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tmp-dir\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.064311 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.064119    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tls-certs\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.064470 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.064425    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.065110 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.064613    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tmp-dir\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.065110 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.064651    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-model-cache\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.065110 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.064867    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-cabundle-cert\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.065110 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.064999    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-home\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.066853 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.066834    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-dshm\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.067266 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.067247    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tls-certs\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.123266 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.123214    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txcrp\" (UniqueName: \"kubernetes.io/projected/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kube-api-access-txcrp\") pod \"lora-single-adapter-test-kserve-78fdc546db-hm4qk\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.187515 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.187486    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:36.613635 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:36.613597    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"]
Oct 30 19:26:36.625774 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:26:36.625748    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0b3df7_d270_42c5_a85a_577c72b2ae1a.slice/crio-c42c0a052b3e4b7dc165292d5f10f99310d2ac958db8ab966ce11eb07e84923d WatchSource:0}: Error finding container c42c0a052b3e4b7dc165292d5f10f99310d2ac958db8ab966ce11eb07e84923d: Status 404 returned error can't find the container with id c42c0a052b3e4b7dc165292d5f10f99310d2ac958db8ab966ce11eb07e84923d
Oct 30 19:26:37.183337 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:37.183304    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" event={"ID":"2f0b3df7-d270-42c5-a85a-577c72b2ae1a","Type":"ContainerStarted","Data":"69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800"}
Oct 30 19:26:37.183878 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:37.183364    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" event={"ID":"2f0b3df7-d270-42c5-a85a-577c72b2ae1a","Type":"ContainerStarted","Data":"c42c0a052b3e4b7dc165292d5f10f99310d2ac958db8ab966ce11eb07e84923d"}
Oct 30 19:26:41.202630 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:41.202596    4853 generic.go:356] "Generic (PLEG): container finished" podID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerID="69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800" exitCode=0
Oct 30 19:26:41.205017 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:41.202677    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" event={"ID":"2f0b3df7-d270-42c5-a85a-577c72b2ae1a","Type":"ContainerDied","Data":"69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800"}
Oct 30 19:26:42.211399 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:42.211366    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"]
Oct 30 19:26:42.212705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:42.212679    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" event={"ID":"2f0b3df7-d270-42c5-a85a-577c72b2ae1a","Type":"ContainerStarted","Data":"f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881"}
Oct 30 19:26:42.282832 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:42.282798    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:42.499794 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:42.499714    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:42.505313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:42.505271    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" podStartSLOduration=3.821289541 podStartE2EDuration="7.505259066s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.683969525s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:42.503727802 +0000 UTC m=+2908.674084685" watchObservedRunningTime="2026-10-30 19:26:42.505259066 +0000 UTC m=+2908.675615914"
Oct 30 19:26:43.187730 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:43.187703    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:46.188166 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:46.188133    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:26:46.189556 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:46.189512    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 30 19:26:49.875308 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:26:49.875264    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:26:56.188478 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:56.188437    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 30 19:26:57.236311 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:57.236255    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main" containerID="cri-o://38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2" gracePeriod=45
Oct 30 19:26:57.847927 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:57.847879    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="llm-d-routing-sidecar" containerID="cri-o://ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b" gracePeriod=45
Oct 30 19:26:58.086142 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.086121    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:26:58.239539 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.239498    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-home\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-home\") pod \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") "
Oct 30 19:26:58.239953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.239555    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-model-cache\") pod \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") "
Oct 30 19:26:58.239953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.239577    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tmp-dir\") pod \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") "
Oct 30 19:26:58.239953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.239599    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-cabundle-cert\") pod \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") "
Oct 30 19:26:58.239953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.239621    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kube-api-access-ql8cw\" (UniqueName: \"kubernetes.io/projected/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kube-api-access-ql8cw\") pod \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") "
Oct 30 19:26:58.239953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.239642    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-dshm\") pod \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") "
Oct 30 19:26:58.239953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.239686    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tls-certs\" (UniqueName: \"kubernetes.io/secret/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tls-certs\") pod \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") "
Oct 30 19:26:58.239953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.239724    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kserve-provision-location\") pod \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\" (UID: \"71b9e4e4-bbe4-485d-a180-a65b2bbd834b\") "
Oct 30 19:26:58.239953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.239906    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-model-cache" pod "71b9e4e4-bbe4-485d-a180-a65b2bbd834b" (UID: "71b9e4e4-bbe4-485d-a180-a65b2bbd834b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:58.240376 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.240144    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-cabundle-cert" pod "71b9e4e4-bbe4-485d-a180-a65b2bbd834b" (UID: "71b9e4e4-bbe4-485d-a180-a65b2bbd834b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:58.240376 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.240220    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-home" pod "71b9e4e4-bbe4-485d-a180-a65b2bbd834b" (UID: "71b9e4e4-bbe4-485d-a180-a65b2bbd834b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:58.242352 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.242322    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tls-certs" pod "71b9e4e4-bbe4-485d-a180-a65b2bbd834b" (UID: "71b9e4e4-bbe4-485d-a180-a65b2bbd834b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:58.242764 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.242744    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kube-api-access-ql8cw" pod "71b9e4e4-bbe4-485d-a180-a65b2bbd834b" (UID: "71b9e4e4-bbe4-485d-a180-a65b2bbd834b"). InnerVolumeSpecName "kube-api-access-ql8cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:58.242851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.242764    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-dshm" pod "71b9e4e4-bbe4-485d-a180-a65b2bbd834b" (UID: "71b9e4e4-bbe4-485d-a180-a65b2bbd834b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:58.254238 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.254213    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tmp-dir" pod "71b9e4e4-bbe4-485d-a180-a65b2bbd834b" (UID: "71b9e4e4-bbe4-485d-a180-a65b2bbd834b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:58.265797 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.265747    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kserve-provision-location" pod "71b9e4e4-bbe4-485d-a180-a65b2bbd834b" (UID: "71b9e4e4-bbe4-485d-a180-a65b2bbd834b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:58.291101 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.291064    4853 generic.go:356] "Generic (PLEG): container finished" podID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerID="38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2" exitCode=0
Oct 30 19:26:58.291194 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.291102    4853 generic.go:356] "Generic (PLEG): container finished" podID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerID="ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b" exitCode=0
Oct 30 19:26:58.291194 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.291139    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"
Oct 30 19:26:58.291194 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.291147    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" event={"ID":"71b9e4e4-bbe4-485d-a180-a65b2bbd834b","Type":"ContainerDied","Data":"38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2"}
Oct 30 19:26:58.291297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.291207    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" event={"ID":"71b9e4e4-bbe4-485d-a180-a65b2bbd834b","Type":"ContainerDied","Data":"ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b"}
Oct 30 19:26:58.291297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.291227    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2" event={"ID":"71b9e4e4-bbe4-485d-a180-a65b2bbd834b","Type":"ContainerDied","Data":"809e88c2629df6bf4dad6da434b1e8d960534e31f9110a0d312f73d0ed807a76"}
Oct 30 19:26:58.291297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.291249    4853 scope.go:122] "RemoveContainer" containerID="38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2"
Oct 30 19:26:58.300278 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.300259    4853 scope.go:122] "RemoveContainer" containerID="967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11"
Oct 30 19:26:58.309313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.309293    4853 scope.go:122] "RemoveContainer" containerID="ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b"
Oct 30 19:26:58.316635 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.316616    4853 scope.go:122] "RemoveContainer" containerID="38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2"
Oct 30 19:26:58.316917 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:26:58.316899    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2\": container with ID starting with 38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2 not found: ID does not exist" containerID="38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2"
Oct 30 19:26:58.316978 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.316926    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2"} err="failed to get container status \"38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2\": rpc error: code = NotFound desc = could not find container \"38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2\": container with ID starting with 38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2 not found: ID does not exist"
Oct 30 19:26:58.316978 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.316943    4853 scope.go:122] "RemoveContainer" containerID="967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11"
Oct 30 19:26:58.317221 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:26:58.317194    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11\": container with ID starting with 967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11 not found: ID does not exist" containerID="967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11"
Oct 30 19:26:58.317296 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.317231    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11"} err="failed to get container status \"967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11\": rpc error: code = NotFound desc = could not find container \"967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11\": container with ID starting with 967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11 not found: ID does not exist"
Oct 30 19:26:58.317296 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.317254    4853 scope.go:122] "RemoveContainer" containerID="ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b"
Oct 30 19:26:58.317500 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:26:58.317478    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b\": container with ID starting with ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b not found: ID does not exist" containerID="ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b"
Oct 30 19:26:58.317563 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.317512    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b"} err="failed to get container status \"ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b\": rpc error: code = NotFound desc = could not find container \"ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b\": container with ID starting with ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b not found: ID does not exist"
Oct 30 19:26:58.317603 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.317564    4853 scope.go:122] "RemoveContainer" containerID="38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2"
Oct 30 19:26:58.317802 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.317779    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2"} err="failed to get container status \"38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2\": rpc error: code = NotFound desc = could not find container \"38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2\": container with ID starting with 38ba5c04d48772bd5b341de58f39a649bd9aab95bedf0d53f6d6195e0bc1a5b2 not found: ID does not exist"
Oct 30 19:26:58.317873 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.317804    4853 scope.go:122] "RemoveContainer" containerID="967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11"
Oct 30 19:26:58.318026 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.318006    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11"} err="failed to get container status \"967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11\": rpc error: code = NotFound desc = could not find container \"967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11\": container with ID starting with 967aaa70d16f7f4a81e8d67722d791cafa3bc9fd72306ea77a38e8b832da6f11 not found: ID does not exist"
Oct 30 19:26:58.318080 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.318026    4853 scope.go:122] "RemoveContainer" containerID="ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b"
Oct 30 19:26:58.318228 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.318213    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b"} err="failed to get container status \"ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b\": rpc error: code = NotFound desc = could not find container \"ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b\": container with ID starting with ac90923459b8ee1fdea29cb8524ff505f19ff4fb623cc01c1d145b0da8b3c58b not found: ID does not exist"
Oct 30 19:26:58.340502 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.340478    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.340598 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.340509    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.340598 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.340544    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.340598 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.340560    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.340598 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.340574    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.340598 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.340586    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql8cw\" (UniqueName: \"kubernetes.io/projected/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-kube-api-access-ql8cw\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.340753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.340600    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.340753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.340614    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b9e4e4-bbe4-485d-a180-a65b2bbd834b-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.453710 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.453682    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"]
Oct 30 19:26:58.460857 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:58.460831    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5944b99c86mpwf2"]
Oct 30 19:26:59.938921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:26:59.938889    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" path="/var/lib/kubelet/pods/71b9e4e4-bbe4-485d-a180-a65b2bbd834b/volumes"
Oct 30 19:27:05.012654 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:05.012622    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:06.188738 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:06.188703    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 30 19:27:16.188395 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:16.188352    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 30 19:27:19.873210 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:27:19.873166    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:27:26.188611 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:26.188573    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 30 19:27:31.796220 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:31.796183    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:32.011384 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:32.011341    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:36.188707 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:36.188667    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 30 19:27:36.834308 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:36.834267    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:37.043177 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:37.043145    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:38.954373 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:38.954333    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:45.420633 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.420597    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"]
Oct 30 19:27:45.421162 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.421145    4853 cpu_manager.go:414] "containerMap: removing container" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="storage-initializer"
Oct 30 19:27:45.421217 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.421165    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="storage-initializer"
Oct 30 19:27:45.421217 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.421177    4853 cpu_manager.go:414] "containerMap: removing container" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="llm-d-routing-sidecar"
Oct 30 19:27:45.421217 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.421185    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="llm-d-routing-sidecar"
Oct 30 19:27:45.421217 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.421209    4853 cpu_manager.go:414] "containerMap: removing container" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main"
Oct 30 19:27:45.421337 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.421218    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main"
Oct 30 19:27:45.421337 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.421285    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="main"
Oct 30 19:27:45.421337 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.421300    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b9e4e4-bbe4-485d-a180-a65b2bbd834b" containerName="llm-d-routing-sidecar"
Oct 30 19:27:45.424479 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.424458    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.461214 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.461184    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"]
Oct 30 19:27:45.480415 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.480386    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Oct 30 19:27:45.480570 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.480544    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:27:45.480644 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.480544    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Oct 30 19:27:45.480644 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.480552    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Oct 30 19:27:45.480724 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.480684    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Oct 30 19:27:45.480770 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.480724    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-fhftj\""
Oct 30 19:27:45.486162 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.486134    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.486280 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.486179    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.486280 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.486212    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.486389 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.486352    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85874352-35fd-40d2-a4ad-20f992c22c6f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.486389 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.486378    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.486497 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.486410    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f62\" (UniqueName: \"kubernetes.io/projected/85874352-35fd-40d2-a4ad-20f992c22c6f-kube-api-access-r4f62\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.486497 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.486432    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.486603 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.486553    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85874352-35fd-40d2-a4ad-20f992c22c6f-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.524421 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.524389    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:45.587865 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.587842    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.588022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.587883    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.588022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.587913    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85874352-35fd-40d2-a4ad-20f992c22c6f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.588022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.587943    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.588022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.587984    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f62\" (UniqueName: \"kubernetes.io/projected/85874352-35fd-40d2-a4ad-20f992c22c6f-kube-api-access-r4f62\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.588022 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.588010    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.589419 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.588051    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85874352-35fd-40d2-a4ad-20f992c22c6f-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.589419 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.588084    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.589419 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.588416    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.589419 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.588523    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.589419 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.588641    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.589419 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.588882    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.589419 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.589113    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85874352-35fd-40d2-a4ad-20f992c22c6f-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.590750 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.590720    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.590996 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.590979    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85874352-35fd-40d2-a4ad-20f992c22c6f-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.643783 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.643753    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f62\" (UniqueName: \"kubernetes.io/projected/85874352-35fd-40d2-a4ad-20f992c22c6f-kube-api-access-r4f62\") pod \"custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:45.742567 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:45.742476    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:46.004208 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:46.004115    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:46.188890 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:46.188855    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 30 19:27:46.225636 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:46.225600    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"]
Oct 30 19:27:46.235760 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:27:46.235716    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85874352_35fd_40d2_a4ad_20f992c22c6f.slice/crio-f1f78349204cfbbced54ee128b2e420f1c5061441dbee2795f0b33cabf7759a9 WatchSource:0}: Error finding container f1f78349204cfbbced54ee128b2e420f1c5061441dbee2795f0b33cabf7759a9: Status 404 returned error can't find the container with id f1f78349204cfbbced54ee128b2e420f1c5061441dbee2795f0b33cabf7759a9
Oct 30 19:27:46.534152 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:46.534056    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" event={"ID":"85874352-35fd-40d2-a4ad-20f992c22c6f","Type":"ContainerStarted","Data":"64a55abcfa840efe640fe4edbf836ecb1980b80f10097c05860cb47bf91f9a22"}
Oct 30 19:27:46.534152 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:46.534113    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" event={"ID":"85874352-35fd-40d2-a4ad-20f992c22c6f","Type":"ContainerStarted","Data":"f1f78349204cfbbced54ee128b2e420f1c5061441dbee2795f0b33cabf7759a9"}
Oct 30 19:27:46.780264 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:46.780227    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:46.994441 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:46.994402    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:49.552633 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:49.552597    4853 generic.go:356] "Generic (PLEG): container finished" podID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerID="64a55abcfa840efe640fe4edbf836ecb1980b80f10097c05860cb47bf91f9a22" exitCode=0
Oct 30 19:27:49.553073 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:49.552655    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" event={"ID":"85874352-35fd-40d2-a4ad-20f992c22c6f","Type":"ContainerDied","Data":"64a55abcfa840efe640fe4edbf836ecb1980b80f10097c05860cb47bf91f9a22"}
Oct 30 19:27:49.884128 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:27:49.884096    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:27:50.562259 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:50.562225    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" event={"ID":"85874352-35fd-40d2-a4ad-20f992c22c6f","Type":"ContainerStarted","Data":"7337b694423de61aacd5b82f5e69b5eb2551b40bc5772b21d293d9f350431fb6"}
Oct 30 19:27:50.774898 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:50.774842    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podStartSLOduration=3.480233881 podStartE2EDuration="5.774829678s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.294595797s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:50.727928187 +0000 UTC m=+2976.898285036" watchObservedRunningTime="2026-10-30 19:27:50.774829678 +0000 UTC m=+2976.945186526"
Oct 30 19:27:55.743265 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:55.743223    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:55.743749 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:55.743307    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:27:55.744690 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:55.744663    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 19:27:56.198090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:56.198061    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:27:56.205866 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:27:56.205838    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:28:02.503050 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:02.503002    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:05.344973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:05.344934    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:05.743336 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:05.743248    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 19:28:14.466811 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:14.466777    4853 scope.go:122] "RemoveContainer" containerID="467b83b203f85d99494b6bd31ce31d990ce3e97ca8d03ea1a8cf97145fd01b9a"
Oct 30 19:28:15.743485 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:15.743442    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 19:28:19.076547 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:19.076497    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:19.872881 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:28:19.872840    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:28:23.437609 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:23.437578    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-78fdc546db-hm4qk_2f0b3df7-d270-42c5-a85a-577c72b2ae1a/storage-initializer/0.log"
Oct 30 19:28:23.560484 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:23.560453    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-78fdc546db-hm4qk_2f0b3df7-d270-42c5-a85a-577c72b2ae1a/main/0.log"
Oct 30 19:28:25.742992 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:25.742958    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 19:28:29.881597 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:29.881559    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"]
Oct 30 19:28:29.977087 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:29.977045    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:35.743389 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:35.743346    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 19:28:44.908306 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:44.908258    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main" containerID="cri-o://f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881" gracePeriod=45
Oct 30 19:28:45.199542 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.199427    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 30 19:28:45.557971 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.557948    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:28:45.570068 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570042    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kserve-provision-location\") pod \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") "
Oct 30 19:28:45.570155 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570098    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-model-cache\") pod \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") "
Oct 30 19:28:45.570155 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570146    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-home\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-home\") pod \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") "
Oct 30 19:28:45.570235 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570214    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-cabundle-cert\") pod \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") "
Oct 30 19:28:45.570283 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570243    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tmp-dir\") pod \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") "
Oct 30 19:28:45.570283 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570275    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tls-certs\") pod \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") "
Oct 30 19:28:45.570373 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570298    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-dshm\") pod \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") "
Oct 30 19:28:45.570373 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570329    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kube-api-access-txcrp\" (UniqueName: \"kubernetes.io/projected/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kube-api-access-txcrp\") pod \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\" (UID: \"2f0b3df7-d270-42c5-a85a-577c72b2ae1a\") "
Oct 30 19:28:45.570373 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570332    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-model-cache" pod "2f0b3df7-d270-42c5-a85a-577c72b2ae1a" (UID: "2f0b3df7-d270-42c5-a85a-577c72b2ae1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:45.570821 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570574    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:45.570821 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570602    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-cabundle-cert" pod "2f0b3df7-d270-42c5-a85a-577c72b2ae1a" (UID: "2f0b3df7-d270-42c5-a85a-577c72b2ae1a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:45.570821 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.570615    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-home" pod "2f0b3df7-d270-42c5-a85a-577c72b2ae1a" (UID: "2f0b3df7-d270-42c5-a85a-577c72b2ae1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:45.572773 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.572744    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tls-certs" pod "2f0b3df7-d270-42c5-a85a-577c72b2ae1a" (UID: "2f0b3df7-d270-42c5-a85a-577c72b2ae1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:45.572866 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.572853    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-dshm" pod "2f0b3df7-d270-42c5-a85a-577c72b2ae1a" (UID: "2f0b3df7-d270-42c5-a85a-577c72b2ae1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:45.573479 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.573454    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kube-api-access-txcrp" pod "2f0b3df7-d270-42c5-a85a-577c72b2ae1a" (UID: "2f0b3df7-d270-42c5-a85a-577c72b2ae1a"). InnerVolumeSpecName "kube-api-access-txcrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:45.582192 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.582165    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tmp-dir" pod "2f0b3df7-d270-42c5-a85a-577c72b2ae1a" (UID: "2f0b3df7-d270-42c5-a85a-577c72b2ae1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:45.594147 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.594128    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kserve-provision-location" pod "2f0b3df7-d270-42c5-a85a-577c72b2ae1a" (UID: "2f0b3df7-d270-42c5-a85a-577c72b2ae1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:28:45.671027 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.671009    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:45.671027 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.671028    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:45.671162 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.671038    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:45.671162 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.671046    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:45.671162 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.671054    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:45.671162 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.671062    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txcrp\" (UniqueName: \"kubernetes.io/projected/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kube-api-access-txcrp\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:45.671162 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.671070    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f0b3df7-d270-42c5-a85a-577c72b2ae1a-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:45.743335 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.743272    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 19:28:45.834423 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.834389    4853 generic.go:356] "Generic (PLEG): container finished" podID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerID="f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881" exitCode=0
Oct 30 19:28:45.834423 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.834423    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" event={"ID":"2f0b3df7-d270-42c5-a85a-577c72b2ae1a","Type":"ContainerDied","Data":"f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881"}
Oct 30 19:28:45.834658 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.834450    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"
Oct 30 19:28:45.834658 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.834464    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk" event={"ID":"2f0b3df7-d270-42c5-a85a-577c72b2ae1a","Type":"ContainerDied","Data":"c42c0a052b3e4b7dc165292d5f10f99310d2ac958db8ab966ce11eb07e84923d"}
Oct 30 19:28:45.834658 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.834485    4853 scope.go:122] "RemoveContainer" containerID="f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881"
Oct 30 19:28:45.844259 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.844232    4853 scope.go:122] "RemoveContainer" containerID="69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800"
Oct 30 19:28:45.852848 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.852831    4853 scope.go:122] "RemoveContainer" containerID="f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881"
Oct 30 19:28:45.853110 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:28:45.853089    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881\": container with ID starting with f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881 not found: ID does not exist" containerID="f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881"
Oct 30 19:28:45.853192 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.853121    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881"} err="failed to get container status \"f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881\": rpc error: code = NotFound desc = could not find container \"f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881\": container with ID starting with f4704e01558dfd60c86f875bf126a193f37051fe6b1b8d8cbc7209d7d4435881 not found: ID does not exist"
Oct 30 19:28:45.853192 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.853148    4853 scope.go:122] "RemoveContainer" containerID="69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800"
Oct 30 19:28:45.853385 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:28:45.853365    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800\": container with ID starting with 69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800 not found: ID does not exist" containerID="69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800"
Oct 30 19:28:45.853431 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.853390    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800"} err="failed to get container status \"69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800\": rpc error: code = NotFound desc = could not find container \"69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800\": container with ID starting with 69a0f3d3da815a785d75f9086354b952f714b3deed4bd3821540a9f902f2f800 not found: ID does not exist"
Oct 30 19:28:45.993262 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.993231    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"]
Oct 30 19:28:45.999302 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:45.999236    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-78fdc546db-hm4qk"]
Oct 30 19:28:47.940291 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:47.940255    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" path="/var/lib/kubelet/pods/2f0b3df7-d270-42c5-a85a-577c72b2ae1a/volumes"
Oct 30 19:28:49.874393 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:28:49.874355    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:28:50.149795 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:50.149711    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:50.376108 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:50.376072    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:50.587083 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:50.587050    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:55.743367 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:55.743322    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 19:28:56.367161 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.367128    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"]
Oct 30 19:28:56.367647 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.367626    4853 cpu_manager.go:414] "containerMap: removing container" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main"
Oct 30 19:28:56.367745 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.367650    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main"
Oct 30 19:28:56.367745 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.367685    4853 cpu_manager.go:414] "containerMap: removing container" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="storage-initializer"
Oct 30 19:28:56.367745 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.367694    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="storage-initializer"
Oct 30 19:28:56.367908 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.367779    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0b3df7-d270-42c5-a85a-577c72b2ae1a" containerName="main"
Oct 30 19:28:56.372906 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.372878    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.417444 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.417416    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"]
Oct 30 19:28:56.441394 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.441364    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-model-cache\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.441518 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.441458    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtfdb\" (UniqueName: \"kubernetes.io/projected/b8db7e66-2b1e-4835-ac5b-45276261980a-kube-api-access-gtfdb\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.441604 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.441511    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-dshm\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.441604 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.441571    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8db7e66-2b1e-4835-ac5b-45276261980a-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.441740 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.441680    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8db7e66-2b1e-4835-ac5b-45276261980a-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.441740 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.441709    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.441806 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.441760    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-home\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.441850 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.441803    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.461739 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.461712    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Oct 30 19:28:56.461739 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.461713    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Oct 30 19:28:56.461910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.461862    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:28:56.461910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.461868    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Oct 30 19:28:56.462001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.461909    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-fwlgb\""
Oct 30 19:28:56.462001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.461928    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Oct 30 19:28:56.479824 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.479793    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:56.542758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.542729    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.542888 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.542816    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-model-cache\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.542888 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.542859    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtfdb\" (UniqueName: \"kubernetes.io/projected/b8db7e66-2b1e-4835-ac5b-45276261980a-kube-api-access-gtfdb\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.542990 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.542894    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-dshm\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.542990 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.542921    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8db7e66-2b1e-4835-ac5b-45276261980a-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.542990 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.542955    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8db7e66-2b1e-4835-ac5b-45276261980a-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.543144 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.542995    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.543144 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.543028    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-home\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.543253 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.543141    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.543253 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.543193    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-model-cache\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.543404 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.543382    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-home\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.543516 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.543489    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.543667 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.543617    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8db7e66-2b1e-4835-ac5b-45276261980a-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.545204 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.545183    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-dshm\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.545567 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.545521    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8db7e66-2b1e-4835-ac5b-45276261980a-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.600100 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.600078    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtfdb\" (UniqueName: \"kubernetes.io/projected/b8db7e66-2b1e-4835-ac5b-45276261980a-kube-api-access-gtfdb\") pod \"lora-multiple-adapters-test-kserve-7975fbd655-hq9g6\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:56.689267 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:56.689197    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:28:57.131187 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:57.131141    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"]
Oct 30 19:28:57.139575 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:28:57.139508    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8db7e66_2b1e_4835_ac5b_45276261980a.slice/crio-b4798737a03d9fd3d732f6ff22228675531cdbfc20411f85a03b5887bfcbba02 WatchSource:0}: Error finding container b4798737a03d9fd3d732f6ff22228675531cdbfc20411f85a03b5887bfcbba02: Status 404 returned error can't find the container with id b4798737a03d9fd3d732f6ff22228675531cdbfc20411f85a03b5887bfcbba02
Oct 30 19:28:57.899923 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:57.899885    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" event={"ID":"b8db7e66-2b1e-4835-ac5b-45276261980a","Type":"ContainerStarted","Data":"19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f"}
Oct 30 19:28:57.900138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:57.899950    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" event={"ID":"b8db7e66-2b1e-4835-ac5b-45276261980a","Type":"ContainerStarted","Data":"b4798737a03d9fd3d732f6ff22228675531cdbfc20411f85a03b5887bfcbba02"}
Oct 30 19:28:57.924385 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:57.924326    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:58.148058 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:28:58.148021    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:01.920166 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:01.920132    4853 generic.go:356] "Generic (PLEG): container finished" podID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerID="19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f" exitCode=0
Oct 30 19:29:01.920653 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:01.920190    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" event={"ID":"b8db7e66-2b1e-4835-ac5b-45276261980a","Type":"ContainerDied","Data":"19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f"}
Oct 30 19:29:02.926536 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:02.926495    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" event={"ID":"b8db7e66-2b1e-4835-ac5b-45276261980a","Type":"ContainerStarted","Data":"920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7"}
Oct 30 19:29:03.136176 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:03.136118    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" podStartSLOduration=2.950109238 podStartE2EDuration="7.136098762s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.185989524s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:03.088009253 +0000 UTC m=+3049.258366127" watchObservedRunningTime="2026-10-30 19:29:03.136098762 +0000 UTC m=+3049.306455617"
Oct 30 19:29:03.690233 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:03.690193    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:29:05.733162 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:05.733127    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:05.743177 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:05.743144    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 19:29:06.690080 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:06.690042    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:29:06.691814 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:06.691784    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 30 19:29:15.743181 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:15.743133    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 19:29:16.690358 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:16.690314    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 30 19:29:19.875437 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:29:19.875397    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:29:25.743516 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:25.743474    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 19:29:26.690239 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:26.690196    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 30 19:29:35.743150 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:35.743090    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 19:29:36.690373 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:36.690325    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 30 19:29:45.752926 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:45.752888    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:29:45.760747 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:45.760712    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:29:46.689788 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:46.689751    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 30 19:29:49.883541 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:29:49.883480    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:29:56.690047 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:56.690003    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 30 19:29:58.554247 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:58.554213    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"]
Oct 30 19:29:58.632920 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:58.632887    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:58.951066 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:29:58.951029    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:06.041420 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:06.041382    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:06.689913 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:06.689866    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 30 19:30:13.579024 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:13.578980    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main" containerID="cri-o://7337b694423de61aacd5b82f5e69b5eb2551b40bc5772b21d293d9f350431fb6" gracePeriod=45
Oct 30 19:30:14.294352 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.294322    4853 generic.go:356] "Generic (PLEG): container finished" podID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerID="7337b694423de61aacd5b82f5e69b5eb2551b40bc5772b21d293d9f350431fb6" exitCode=0
Oct 30 19:30:14.294486 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.294389    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" event={"ID":"85874352-35fd-40d2-a4ad-20f992c22c6f","Type":"ContainerDied","Data":"7337b694423de61aacd5b82f5e69b5eb2551b40bc5772b21d293d9f350431fb6"}
Oct 30 19:30:14.316738 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.316719    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:30:14.369893 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.369841    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85874352-35fd-40d2-a4ad-20f992c22c6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/85874352-35fd-40d2-a4ad-20f992c22c6f-tls-certs\") pod \"85874352-35fd-40d2-a4ad-20f992c22c6f\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") "
Oct 30 19:30:14.369980 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.369891    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-tmp-dir\") pod \"85874352-35fd-40d2-a4ad-20f992c22c6f\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") "
Oct 30 19:30:14.370019 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.369982    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85874352-35fd-40d2-a4ad-20f992c22c6f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85874352-35fd-40d2-a4ad-20f992c22c6f-cabundle-cert\") pod \"85874352-35fd-40d2-a4ad-20f992c22c6f\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") "
Oct 30 19:30:14.370056 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.370015    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-model-cache\") pod \"85874352-35fd-40d2-a4ad-20f992c22c6f\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") "
Oct 30 19:30:14.370097 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.370056    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-kserve-provision-location\") pod \"85874352-35fd-40d2-a4ad-20f992c22c6f\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") "
Oct 30 19:30:14.370097 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.370087    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-dshm\") pod \"85874352-35fd-40d2-a4ad-20f992c22c6f\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") "
Oct 30 19:30:14.370175 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.370112    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85874352-35fd-40d2-a4ad-20f992c22c6f-kube-api-access-r4f62\" (UniqueName: \"kubernetes.io/projected/85874352-35fd-40d2-a4ad-20f992c22c6f-kube-api-access-r4f62\") pod \"85874352-35fd-40d2-a4ad-20f992c22c6f\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") "
Oct 30 19:30:14.370175 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.370148    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-home\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-home\") pod \"85874352-35fd-40d2-a4ad-20f992c22c6f\" (UID: \"85874352-35fd-40d2-a4ad-20f992c22c6f\") "
Oct 30 19:30:14.370347 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.370322    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85874352-35fd-40d2-a4ad-20f992c22c6f-cabundle-cert" pod "85874352-35fd-40d2-a4ad-20f992c22c6f" (UID: "85874352-35fd-40d2-a4ad-20f992c22c6f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:14.370464 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.370403    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-model-cache" pod "85874352-35fd-40d2-a4ad-20f992c22c6f" (UID: "85874352-35fd-40d2-a4ad-20f992c22c6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:14.370877 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.370859    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-home" pod "85874352-35fd-40d2-a4ad-20f992c22c6f" (UID: "85874352-35fd-40d2-a4ad-20f992c22c6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:14.372430 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.372397    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85874352-35fd-40d2-a4ad-20f992c22c6f-tls-certs" pod "85874352-35fd-40d2-a4ad-20f992c22c6f" (UID: "85874352-35fd-40d2-a4ad-20f992c22c6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:14.372659 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.372636    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-dshm" pod "85874352-35fd-40d2-a4ad-20f992c22c6f" (UID: "85874352-35fd-40d2-a4ad-20f992c22c6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:14.372812 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.372795    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85874352-35fd-40d2-a4ad-20f992c22c6f-kube-api-access-r4f62" pod "85874352-35fd-40d2-a4ad-20f992c22c6f" (UID: "85874352-35fd-40d2-a4ad-20f992c22c6f"). InnerVolumeSpecName "kube-api-access-r4f62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:14.389597 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.389564    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-tmp-dir" pod "85874352-35fd-40d2-a4ad-20f992c22c6f" (UID: "85874352-35fd-40d2-a4ad-20f992c22c6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:14.397039 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.397017    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-kserve-provision-location" pod "85874352-35fd-40d2-a4ad-20f992c22c6f" (UID: "85874352-35fd-40d2-a4ad-20f992c22c6f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:14.470821 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.470799    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85874352-35fd-40d2-a4ad-20f992c22c6f-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.470821 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.470821    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.470953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.470831    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.470953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.470840    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.470953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.470849    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4f62\" (UniqueName: \"kubernetes.io/projected/85874352-35fd-40d2-a4ad-20f992c22c6f-kube-api-access-r4f62\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.470953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.470856    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.470953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.470864    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85874352-35fd-40d2-a4ad-20f992c22c6f-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.470953 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:14.470871    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85874352-35fd-40d2-a4ad-20f992c22c6f-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:15.300705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:15.300670    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"
Oct 30 19:30:15.301228 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:15.300669    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj" event={"ID":"85874352-35fd-40d2-a4ad-20f992c22c6f","Type":"ContainerDied","Data":"f1f78349204cfbbced54ee128b2e420f1c5061441dbee2795f0b33cabf7759a9"}
Oct 30 19:30:15.301228 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:15.300813    4853 scope.go:122] "RemoveContainer" containerID="7337b694423de61aacd5b82f5e69b5eb2551b40bc5772b21d293d9f350431fb6"
Oct 30 19:30:15.311194 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:15.311173    4853 scope.go:122] "RemoveContainer" containerID="64a55abcfa840efe640fe4edbf836ecb1980b80f10097c05860cb47bf91f9a22"
Oct 30 19:30:15.507732 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:15.507700    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"]
Oct 30 19:30:15.553941 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:15.553877    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7495d647c5-2hlgj"]
Oct 30 19:30:15.938888 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:15.938858    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" path="/var/lib/kubelet/pods/85874352-35fd-40d2-a4ad-20f992c22c6f/volumes"
Oct 30 19:30:16.699278 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:16.699244    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:30:16.706878 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:16.706855    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:30:19.872646 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:30:19.872612    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:30:44.551592 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:44.549062    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-7975fbd655-hq9g6_b8db7e66-2b1e-4835-ac5b-45276261980a/storage-initializer/0.log"
Oct 30 19:30:44.669818 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:44.669789    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-7975fbd655-hq9g6_b8db7e66-2b1e-4835-ac5b-45276261980a/main/0.log"
Oct 30 19:30:48.197574 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:48.197495    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:48.423621 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:48.423584    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:48.821961 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:48.821925    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:49.038402 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:49.038320    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:49.492099 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:49.492067    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"]
Oct 30 19:30:49.569860 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:49.569830    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:49.873032 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:30:49.872998    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:30:52.597250 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:52.597217    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:52.958628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:52.958591    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:55.978570 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:30:55.978518    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:00.625133 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.625103    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"]
Oct 30 19:31:00.625522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.625454    4853 cpu_manager.go:414] "containerMap: removing container" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="storage-initializer"
Oct 30 19:31:00.625522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.625466    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="storage-initializer"
Oct 30 19:31:00.625522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.625482    4853 cpu_manager.go:414] "containerMap: removing container" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main"
Oct 30 19:31:00.625522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.625487    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main"
Oct 30 19:31:00.625674 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.625569    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="85874352-35fd-40d2-a4ad-20f992c22c6f" containerName="main"
Oct 30 19:31:00.628766 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.628751    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.639365 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.639344    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"]
Oct 30 19:31:00.679176 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.679143    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Oct 30 19:31:00.683755 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.683729    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:31:00.683932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.683733    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Oct 30 19:31:00.683932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.683740    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Oct 30 19:31:00.683932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.683753    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Oct 30 19:31:00.684169 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.683820    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-rk77l\""
Oct 30 19:31:00.716719 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.716687    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:00.718829 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.718808    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660f0ed2-bac3-4b78-90ee-b0546b949747-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.718891 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.718843    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54h87\" (UniqueName: \"kubernetes.io/projected/660f0ed2-bac3-4b78-90ee-b0546b949747-kube-api-access-54h87\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.718933 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.718913    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660f0ed2-bac3-4b78-90ee-b0546b949747-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.718998 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.718986    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.719038 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.719015    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.719077 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.719051    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.719077 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.719069    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-home\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.719152 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.719118    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.820400 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.820368    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660f0ed2-bac3-4b78-90ee-b0546b949747-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.820572 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.820484    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.820572 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.820516    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.820672 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.820581    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.820672 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.820601    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-home\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.820672 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.820618    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.820672 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.820646    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660f0ed2-bac3-4b78-90ee-b0546b949747-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.820672 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.820669    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54h87\" (UniqueName: \"kubernetes.io/projected/660f0ed2-bac3-4b78-90ee-b0546b949747-kube-api-access-54h87\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.820940 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.820908    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.821045 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.821027    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.821093 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.821024    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.821143 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.821116    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660f0ed2-bac3-4b78-90ee-b0546b949747-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.821143 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.821128    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-home\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.823094 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.823074    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.823371 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.823352    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660f0ed2-bac3-4b78-90ee-b0546b949747-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.876804 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.876744    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54h87\" (UniqueName: \"kubernetes.io/projected/660f0ed2-bac3-4b78-90ee-b0546b949747-kube-api-access-54h87\") pod \"router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:00.945690 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:00.945663    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:01.186569 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:01.186478    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:01.376266 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:01.376234    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"]
Oct 30 19:31:01.384238 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:31:01.384195    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660f0ed2_bac3_4b78_90ee_b0546b949747.slice/crio-1e23336669017ffde55ea945c89b4ce4e0173a810ccfbbc5e547f59a3ddf7c0c WatchSource:0}: Error finding container 1e23336669017ffde55ea945c89b4ce4e0173a810ccfbbc5e547f59a3ddf7c0c: Status 404 returned error can't find the container with id 1e23336669017ffde55ea945c89b4ce4e0173a810ccfbbc5e547f59a3ddf7c0c
Oct 30 19:31:01.388795 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:01.388755    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:01.509496 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:01.509466    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" event={"ID":"660f0ed2-bac3-4b78-90ee-b0546b949747","Type":"ContainerStarted","Data":"228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83"}
Oct 30 19:31:01.509651 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:01.509516    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" event={"ID":"660f0ed2-bac3-4b78-90ee-b0546b949747","Type":"ContainerStarted","Data":"1e23336669017ffde55ea945c89b4ce4e0173a810ccfbbc5e547f59a3ddf7c0c"}
Oct 30 19:31:03.290042 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:03.290012    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:03.508375 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:03.508339    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:04.514853 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:04.514813    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main" containerID="cri-o://920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7" gracePeriod=45
Oct 30 19:31:04.525653 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:04.525625    4853 generic.go:356] "Generic (PLEG): container finished" podID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerID="228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83" exitCode=0
Oct 30 19:31:04.525779 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:04.525671    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" event={"ID":"660f0ed2-bac3-4b78-90ee-b0546b949747","Type":"ContainerDied","Data":"228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83"}
Oct 30 19:31:04.700568 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:04.700506    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 30 19:31:05.154727 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.154691    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:31:05.260933 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.260221    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-kserve-provision-location\") pod \"b8db7e66-2b1e-4835-ac5b-45276261980a\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") "
Oct 30 19:31:05.260933 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.260294    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-dshm\") pod \"b8db7e66-2b1e-4835-ac5b-45276261980a\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") "
Oct 30 19:31:05.260933 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.260324    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8db7e66-2b1e-4835-ac5b-45276261980a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8db7e66-2b1e-4835-ac5b-45276261980a-tls-certs\") pod \"b8db7e66-2b1e-4835-ac5b-45276261980a\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") "
Oct 30 19:31:05.260933 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.260384    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-tmp-dir\") pod \"b8db7e66-2b1e-4835-ac5b-45276261980a\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") "
Oct 30 19:31:05.260933 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.260418    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8db7e66-2b1e-4835-ac5b-45276261980a-kube-api-access-gtfdb\" (UniqueName: \"kubernetes.io/projected/b8db7e66-2b1e-4835-ac5b-45276261980a-kube-api-access-gtfdb\") pod \"b8db7e66-2b1e-4835-ac5b-45276261980a\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") "
Oct 30 19:31:05.260933 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.260446    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8db7e66-2b1e-4835-ac5b-45276261980a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8db7e66-2b1e-4835-ac5b-45276261980a-cabundle-cert\") pod \"b8db7e66-2b1e-4835-ac5b-45276261980a\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") "
Oct 30 19:31:05.260933 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.260500    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-model-cache\") pod \"b8db7e66-2b1e-4835-ac5b-45276261980a\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") "
Oct 30 19:31:05.260933 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.260559    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-home\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-home\") pod \"b8db7e66-2b1e-4835-ac5b-45276261980a\" (UID: \"b8db7e66-2b1e-4835-ac5b-45276261980a\") "
Oct 30 19:31:05.261738 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.261600    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-home" pod "b8db7e66-2b1e-4835-ac5b-45276261980a" (UID: "b8db7e66-2b1e-4835-ac5b-45276261980a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:05.262647 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.262386    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8db7e66-2b1e-4835-ac5b-45276261980a-cabundle-cert" pod "b8db7e66-2b1e-4835-ac5b-45276261980a" (UID: "b8db7e66-2b1e-4835-ac5b-45276261980a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:05.263842 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.263802    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-model-cache" pod "b8db7e66-2b1e-4835-ac5b-45276261980a" (UID: "b8db7e66-2b1e-4835-ac5b-45276261980a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:05.264592 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.264566    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8db7e66-2b1e-4835-ac5b-45276261980a-kube-api-access-gtfdb" pod "b8db7e66-2b1e-4835-ac5b-45276261980a" (UID: "b8db7e66-2b1e-4835-ac5b-45276261980a"). InnerVolumeSpecName "kube-api-access-gtfdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:05.265056 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.265034    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-dshm" pod "b8db7e66-2b1e-4835-ac5b-45276261980a" (UID: "b8db7e66-2b1e-4835-ac5b-45276261980a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:05.265671 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.265647    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8db7e66-2b1e-4835-ac5b-45276261980a-tls-certs" pod "b8db7e66-2b1e-4835-ac5b-45276261980a" (UID: "b8db7e66-2b1e-4835-ac5b-45276261980a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:05.277870 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.277845    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-tmp-dir" pod "b8db7e66-2b1e-4835-ac5b-45276261980a" (UID: "b8db7e66-2b1e-4835-ac5b-45276261980a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:05.286825 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.286801    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-kserve-provision-location" pod "b8db7e66-2b1e-4835-ac5b-45276261980a" (UID: "b8db7e66-2b1e-4835-ac5b-45276261980a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:05.362214 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.362182    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.362307 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.362220    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8db7e66-2b1e-4835-ac5b-45276261980a-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.362307 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.362234    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.362307 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.362247    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtfdb\" (UniqueName: \"kubernetes.io/projected/b8db7e66-2b1e-4835-ac5b-45276261980a-kube-api-access-gtfdb\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.362307 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.362263    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8db7e66-2b1e-4835-ac5b-45276261980a-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.362307 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.362278    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.362307 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.362291    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.362307 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.362304    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8db7e66-2b1e-4835-ac5b-45276261980a-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:05.534024 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.533941    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" event={"ID":"660f0ed2-bac3-4b78-90ee-b0546b949747","Type":"ContainerStarted","Data":"aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa"}
Oct 30 19:31:05.535791 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.535763    4853 generic.go:356] "Generic (PLEG): container finished" podID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerID="920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7" exitCode=0
Oct 30 19:31:05.535910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.535807    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" event={"ID":"b8db7e66-2b1e-4835-ac5b-45276261980a","Type":"ContainerDied","Data":"920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7"}
Oct 30 19:31:05.535910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.535842    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6" event={"ID":"b8db7e66-2b1e-4835-ac5b-45276261980a","Type":"ContainerDied","Data":"b4798737a03d9fd3d732f6ff22228675531cdbfc20411f85a03b5887bfcbba02"}
Oct 30 19:31:05.535910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.535863    4853 scope.go:122] "RemoveContainer" containerID="920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7"
Oct 30 19:31:05.535910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.535810    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"
Oct 30 19:31:05.547001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.546976    4853 scope.go:122] "RemoveContainer" containerID="19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f"
Oct 30 19:31:05.557256 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.557228    4853 scope.go:122] "RemoveContainer" containerID="920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7"
Oct 30 19:31:05.557602 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:31:05.557578    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7\": container with ID starting with 920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7 not found: ID does not exist" containerID="920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7"
Oct 30 19:31:05.557691 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.557607    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7"} err="failed to get container status \"920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7\": rpc error: code = NotFound desc = could not find container \"920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7\": container with ID starting with 920844b52fe4bcdad44b634b186a6f5bc9832412f1fb5598990ffee2dbb937c7 not found: ID does not exist"
Oct 30 19:31:05.557691 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.557629    4853 scope.go:122] "RemoveContainer" containerID="19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f"
Oct 30 19:31:05.557905 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:31:05.557887    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f\": container with ID starting with 19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f not found: ID does not exist" containerID="19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f"
Oct 30 19:31:05.557951 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.557918    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f"} err="failed to get container status \"19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f\": rpc error: code = NotFound desc = could not find container \"19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f\": container with ID starting with 19b79a120ff80a3f16313ffbc4988666e83ce5d991648fdd06f0eb4c9db5131f not found: ID does not exist"
Oct 30 19:31:05.652189 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.652141    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podStartSLOduration=3.511136832 podStartE2EDuration="5.652128141s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.140991309s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:05.650143329 +0000 UTC m=+3171.820500178" watchObservedRunningTime="2026-10-30 19:31:05.652128141 +0000 UTC m=+3171.822484989"
Oct 30 19:31:05.759330 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.759296    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"]
Oct 30 19:31:05.763968 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.763944    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7975fbd655-hq9g6"]
Oct 30 19:31:05.939158 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:05.939109    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" path="/var/lib/kubelet/pods/b8db7e66-2b1e-4835-ac5b-45276261980a/volumes"
Oct 30 19:31:06.365059 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:06.365028    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:09.788964 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:09.788926    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:09.994174 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:09.994136    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:10.262704 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:10.262672    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:10.491416 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:10.491379    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:10.692706 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:10.692677    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:10.946747 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:10.946655    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:10.946747 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:10.946715    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:31:10.948506 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:10.948480    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Oct 30 19:31:13.384054 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.384016    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"]
Oct 30 19:31:13.384430 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.384314    4853 cpu_manager.go:414] "containerMap: removing container" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main"
Oct 30 19:31:13.384430 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.384325    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main"
Oct 30 19:31:13.384430 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.384337    4853 cpu_manager.go:414] "containerMap: removing container" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="storage-initializer"
Oct 30 19:31:13.384430 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.384343    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="storage-initializer"
Oct 30 19:31:13.384430 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.384412    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8db7e66-2b1e-4835-ac5b-45276261980a" containerName="main"
Oct 30 19:31:13.399935 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.399912    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:13.440349 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.440325    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"]
Oct 30 19:31:13.465678 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.465656    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:13.485624 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.485601    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 30 19:31:13.485624 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.485603    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Oct 30 19:31:13.485764 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.485608    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-ldp4s\""
Oct 30 19:31:13.496966 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.496943    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Oct 30 19:31:13.627736 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.627707    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qtfh\" (UniqueName: \"kubernetes.io/projected/dd9f8e0f-67c5-4614-b210-b9d3ff3306b4-kube-api-access-5qtfh\") pod \"e2e-pvc-lora-download-ghl67\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:13.669442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.669375    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:13.728311 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.728290    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qtfh\" (UniqueName: \"kubernetes.io/projected/dd9f8e0f-67c5-4614-b210-b9d3ff3306b4-kube-api-access-5qtfh\") pod \"e2e-pvc-lora-download-ghl67\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:13.784951 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:13.784921    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qtfh\" (UniqueName: \"kubernetes.io/projected/dd9f8e0f-67c5-4614-b210-b9d3ff3306b4-kube-api-access-5qtfh\") pod \"e2e-pvc-lora-download-ghl67\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:15.845793 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:15.845761    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"e2e-pvc-lora-download-ghl67\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:15.944018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:15.943986    4853 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"e2e-pvc-lora-download-ghl67\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:15.946761 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:15.946737    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"e2e-pvc-lora-download-ghl67\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:15.946878 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:15.946844    4853 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"e2e-pvc-lora-download-ghl67\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:15.998397 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:15.998371    4853 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"e2e-pvc-lora-download-ghl67\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") DevicePath \"csi-629954111aa2ac0ba579bcd4565a19ffaf5549c615d1b4d7092ec334191f49c9\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:16.259145 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:16.259098    4853 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"e2e-pvc-lora-download-ghl67\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7ac11c389c5cc070e07feee4498c0a13da71b0ffa76479b4ccf19e65880bb82a/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:16.319971 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:16.319940    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"e2e-pvc-lora-download-ghl67\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:16.458604 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:16.458573    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-ldp4s\""
Oct 30 19:31:16.460583 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:16.460565    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:16.834775 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:16.834745    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"]
Oct 30 19:31:16.841427 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:31:16.841396    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd9f8e0f_67c5_4614_b210_b9d3ff3306b4.slice/crio-394eaf0b0fd6d331f57a99d8dfdc416c0152fea7c0be741e0a1dd17eee92333d WatchSource:0}: Error finding container 394eaf0b0fd6d331f57a99d8dfdc416c0152fea7c0be741e0a1dd17eee92333d: Status 404 returned error can't find the container with id 394eaf0b0fd6d331f57a99d8dfdc416c0152fea7c0be741e0a1dd17eee92333d
Oct 30 19:31:16.891374 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:16.891349    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 30 19:31:16.939380 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:16.939357    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:17.164516 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:17.164431    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:17.598436 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:17.598397    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67" event={"ID":"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4","Type":"ContainerStarted","Data":"dc91f6bab4f37bc2e165dc6472762786dab8a59c9f6346ad22cbc30a0e7a7aae"}
Oct 30 19:31:17.598651 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:17.598449    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67" event={"ID":"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4","Type":"ContainerStarted","Data":"394eaf0b0fd6d331f57a99d8dfdc416c0152fea7c0be741e0a1dd17eee92333d"}
Oct 30 19:31:17.709041 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:17.708986    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67" podStartSLOduration=7.708971928 podStartE2EDuration="7.708971928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:17.707975062 +0000 UTC m=+3183.878331910" watchObservedRunningTime="2026-10-30 19:31:17.708971928 +0000 UTC m=+3183.879328773"
Oct 30 19:31:19.610641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:19.610557    4853 generic.go:356] "Generic (PLEG): container finished" podID="dd9f8e0f-67c5-4614-b210-b9d3ff3306b4" containerID="dc91f6bab4f37bc2e165dc6472762786dab8a59c9f6346ad22cbc30a0e7a7aae" exitCode=0
Oct 30 19:31:19.611079 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:19.610633    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67" event={"ID":"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4","Type":"ContainerDied","Data":"dc91f6bab4f37bc2e165dc6472762786dab8a59c9f6346ad22cbc30a0e7a7aae"}
Oct 30 19:31:19.873332 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:31:19.873251    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:31:20.742986 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:20.742959    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:20.886469 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:20.886398    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") "
Oct 30 19:31:20.886469 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:20.886449    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9f8e0f-67c5-4614-b210-b9d3ff3306b4-kube-api-access-5qtfh\" (UniqueName: \"kubernetes.io/projected/dd9f8e0f-67c5-4614-b210-b9d3ff3306b4-kube-api-access-5qtfh\") pod \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\" (UID: \"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4\") "
Oct 30 19:31:20.889011 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:20.888982    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9f8e0f-67c5-4614-b210-b9d3ff3306b4-kube-api-access-5qtfh" pod "dd9f8e0f-67c5-4614-b210-b9d3ff3306b4" (UID: "dd9f8e0f-67c5-4614-b210-b9d3ff3306b4"). InnerVolumeSpecName "kube-api-access-5qtfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:20.890411 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:20.890389    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d" pod "dd9f8e0f-67c5-4614-b210-b9d3ff3306b4" (UID: "dd9f8e0f-67c5-4614-b210-b9d3ff3306b4"). InnerVolumeSpecName "pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 19:31:20.946011 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:20.945977    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Oct 30 19:31:20.987360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:20.987331    4853 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" "
Oct 30 19:31:20.987360 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:20.987359    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qtfh\" (UniqueName: \"kubernetes.io/projected/dd9f8e0f-67c5-4614-b210-b9d3ff3306b4-kube-api-access-5qtfh\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:21.620489 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:21.620462    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"
Oct 30 19:31:21.620696 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:21.620455    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67" event={"ID":"dd9f8e0f-67c5-4614-b210-b9d3ff3306b4","Type":"ContainerDied","Data":"394eaf0b0fd6d331f57a99d8dfdc416c0152fea7c0be741e0a1dd17eee92333d"}
Oct 30 19:31:21.620696 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:21.620585    4853 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="394eaf0b0fd6d331f57a99d8dfdc416c0152fea7c0be741e0a1dd17eee92333d"
Oct 30 19:31:21.959040 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:21.959016    4853 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d") on node "ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 19:31:21.995221 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:21.995196    4853 reconciler_common.go:299] "Volume detached for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"csi-629954111aa2ac0ba579bcd4565a19ffaf5549c615d1b4d7092ec334191f49c9\""
Oct 30 19:31:30.946587 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:30.946544    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Oct 30 19:31:40.112775 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.112738    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"]
Oct 30 19:31:40.113224 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.113207    4853 cpu_manager.go:414] "containerMap: removing container" podUID="dd9f8e0f-67c5-4614-b210-b9d3ff3306b4" containerName="storage-initializer"
Oct 30 19:31:40.113267 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.113227    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9f8e0f-67c5-4614-b210-b9d3ff3306b4" containerName="storage-initializer"
Oct 30 19:31:40.113326 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.113312    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9f8e0f-67c5-4614-b210-b9d3ff3306b4" containerName="storage-initializer"
Oct 30 19:31:40.124229 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.124203    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.162276 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.162248    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"]
Oct 30 19:31:40.206846 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.206810    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Oct 30 19:31:40.206846 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.206813    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:31:40.207122 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.206918    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 30 19:31:40.207122 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.206999    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Oct 30 19:31:40.207122 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.207063    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Oct 30 19:31:40.207122 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.207096    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-ldp4s\""
Oct 30 19:31:40.227586 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.227555    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b747f6b5-a02f-4c94-a97c-a215dc31e631-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.227711 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.227605    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b747f6b5-a02f-4c94-a97c-a215dc31e631-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.227711 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.227637    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.227819 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.227794    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.227861 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.227844    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-home\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.227913 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.227897    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.227959 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.227928    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.255152 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.255114    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:40.328879 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.328851    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b747f6b5-a02f-4c94-a97c-a215dc31e631-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329070 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.328886    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329070 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.328924    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksn9\" (UniqueName: \"kubernetes.io/projected/b747f6b5-a02f-4c94-a97c-a215dc31e631-kube-api-access-mksn9\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329070 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.328946    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329070 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.329017    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-home\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329070 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.329039    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329070 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.329057    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329388 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.329114    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b747f6b5-a02f-4c94-a97c-a215dc31e631-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329449 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.329384    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329511 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.329458    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329589 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.329509    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-home\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329692 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.329664    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.329904 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.329884    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b747f6b5-a02f-4c94-a97c-a215dc31e631-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.331662 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.331637    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.331885 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.331866    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b747f6b5-a02f-4c94-a97c-a215dc31e631-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.430639 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.430199    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksn9\" (UniqueName: \"kubernetes.io/projected/b747f6b5-a02f-4c94-a97c-a215dc31e631-kube-api-access-mksn9\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.485937 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.485905    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksn9\" (UniqueName: \"kubernetes.io/projected/b747f6b5-a02f-4c94-a97c-a215dc31e631-kube-api-access-mksn9\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:40.946573 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:40.946514    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Oct 30 19:31:43.258262 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:43.258229    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:45.370431 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:45.370397    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:45.470579 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:45.470547    4853 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:45.470786 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:45.470772    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:45.470868 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:45.470856    4853 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:45.520979 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:45.520953    4853 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") DevicePath \"csi-629954111aa2ac0ba579bcd4565a19ffaf5549c615d1b4d7092ec334191f49c9\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:45.658388 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:45.658353    4853 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7ac11c389c5cc070e07feee4498c0a13da71b0ffa76479b4ccf19e65880bb82a/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:45.724335 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:45.724298    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:45.842623 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:45.842590    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:46.321751 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:46.321720    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"]
Oct 30 19:31:46.329257 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:31:46.329228    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb747f6b5_a02f_4c94_a97c_a215dc31e631.slice/crio-398172d349a93562363d34003c82a17ca1cbec2b10fdc82c9a35f5a59ede57b3 WatchSource:0}: Error finding container 398172d349a93562363d34003c82a17ca1cbec2b10fdc82c9a35f5a59ede57b3: Status 404 returned error can't find the container with id 398172d349a93562363d34003c82a17ca1cbec2b10fdc82c9a35f5a59ede57b3
Oct 30 19:31:46.738256 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:46.738217    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" event={"ID":"b747f6b5-a02f-4c94-a97c-a215dc31e631","Type":"ContainerStarted","Data":"2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6"}
Oct 30 19:31:46.738792 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:46.738280    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" event={"ID":"b747f6b5-a02f-4c94-a97c-a215dc31e631","Type":"ContainerStarted","Data":"398172d349a93562363d34003c82a17ca1cbec2b10fdc82c9a35f5a59ede57b3"}
Oct 30 19:31:48.749447 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:48.749413    4853 generic.go:356] "Generic (PLEG): container finished" podID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerID="2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6" exitCode=0
Oct 30 19:31:48.749887 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:48.749481    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" event={"ID":"b747f6b5-a02f-4c94-a97c-a215dc31e631","Type":"ContainerDied","Data":"2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6"}
Oct 30 19:31:49.757717 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:49.757680    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" event={"ID":"b747f6b5-a02f-4c94-a97c-a215dc31e631","Type":"ContainerStarted","Data":"e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3"}
Oct 30 19:31:49.843754 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:49.843717    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:49.874297 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:31:49.874263    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:31:49.968320 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:49.968255    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" podStartSLOduration=7.690138209 podStartE2EDuration="9.968236022s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.278097813s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:49.920155759 +0000 UTC m=+3216.090512608" watchObservedRunningTime="2026-10-30 19:31:49.968236022 +0000 UTC m=+3216.138592871"
Oct 30 19:31:50.946156 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:50.946111    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Oct 30 19:31:55.547676 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:55.547640    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:55.843736 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:55.843698    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:31:55.845073 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:31:55.845047    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 30 19:32:00.947178 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:00.947117    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Oct 30 19:32:05.843326 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:05.843281    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 30 19:32:06.684011 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.683973    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"]
Oct 30 19:32:06.688907 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.688884    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.739425 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.739389    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-dstgc\""
Oct 30 19:32:06.744749 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.744725    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.744832 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.744791    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7bb60a-962a-401b-8cd9-413217ba989c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.744886 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.744854    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.744925 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.744899    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.744971 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.744927    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.744971 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.744958    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.745068 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.745047    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.745107 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.745072    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vvwl\" (UniqueName: \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-kube-api-access-2vvwl\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.745107 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.745090    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.787616 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.787563    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"]
Oct 30 19:32:06.845805 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.845767    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.846392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.845826    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.846392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.845863    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.846392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.845911    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.846392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.845948    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.846392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.846004    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vvwl\" (UniqueName: \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-kube-api-access-2vvwl\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.846392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.846031    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.846392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.846059    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.846392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.846108    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7bb60a-962a-401b-8cd9-413217ba989c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.846392 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.846290    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.847278 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.846765    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.847278 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.846792    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.847278 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.846993    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.847278 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.847268    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7bb60a-962a-401b-8cd9-413217ba989c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.849323 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.849294    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.849453 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.849291    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.907611 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.903407    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:06.910891 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:06.910859    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vvwl\" (UniqueName: \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-kube-api-access-2vvwl\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9g8z7\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:07.002890 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:07.002798    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:07.436576 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:07.436510    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"]
Oct 30 19:32:07.443167 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:32:07.443121    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7bb60a_962a_401b_8cd9_413217ba989c.slice/crio-b811f0c0144206f4c515462ae9c87c23b0473fec8fef8796eb28d2612b2dfeff WatchSource:0}: Error finding container b811f0c0144206f4c515462ae9c87c23b0473fec8fef8796eb28d2612b2dfeff: Status 404 returned error can't find the container with id b811f0c0144206f4c515462ae9c87c23b0473fec8fef8796eb28d2612b2dfeff
Oct 30 19:32:07.445236 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:07.445200    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:32:07.445330 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:07.445273    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:32:07.445330 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:07.445304    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:32:07.872553 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:07.872484    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" event={"ID":"7c7bb60a-962a-401b-8cd9-413217ba989c","Type":"ContainerStarted","Data":"ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e"}
Oct 30 19:32:07.873059 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:07.872572    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" event={"ID":"7c7bb60a-962a-401b-8cd9-413217ba989c","Type":"ContainerStarted","Data":"b811f0c0144206f4c515462ae9c87c23b0473fec8fef8796eb28d2612b2dfeff"}
Oct 30 19:32:07.982322 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:07.982285    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:08.003345 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:08.003320    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:08.004643 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:08.004615    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:32:08.004782 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:08.004671    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:32:08.082919 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:08.082865    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" podStartSLOduration=2.082850352 podStartE2EDuration="2.082850352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:08.035641842 +0000 UTC m=+3234.205998692" watchObservedRunningTime="2026-10-30 19:32:08.082850352 +0000 UTC m=+3234.253207256"
Oct 30 19:32:09.003345 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:09.003304    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:32:09.003892 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:09.003352    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:32:10.004047 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:10.004015    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:32:10.004608 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:10.004064    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:32:10.948554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:10.948026    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Oct 30 19:32:11.003577 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:11.003519    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:32:11.003768 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:11.003589    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:32:12.003175 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:12.003139    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:32:12.003601 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:12.003204    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:32:13.003693 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:13.003660    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:32:13.004079 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:13.003715    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 19:32:14.008981 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:14.008950    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:14.009463 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:14.009218    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:14.009977 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:14.009958    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:32:15.843079 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:15.843031    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 30 19:32:19.872512 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:32:19.872473    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:32:20.946616 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:20.946575    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Oct 30 19:32:21.816837 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.816799    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"]
Oct 30 19:32:21.820298 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.820275    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.891901 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.891867    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.891901 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.891901    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.892091 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.891929    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.892091 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.892020    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dc1cff-278b-4772-b4ef-661fc700bce7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.892091 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.892041    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.892212 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.892164    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz9fq\" (UniqueName: \"kubernetes.io/projected/82dc1cff-278b-4772-b4ef-661fc700bce7-kube-api-access-kz9fq\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.892250 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.892205    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.892250 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.892239    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.892383 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.892367    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.993412 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.993375    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.993412 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.993412    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.993973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.993437    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.993973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.993466    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dc1cff-278b-4772-b4ef-661fc700bce7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.993973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.993496    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.993973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.993580    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9fq\" (UniqueName: \"kubernetes.io/projected/82dc1cff-278b-4772-b4ef-661fc700bce7-kube-api-access-kz9fq\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.993973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.993607    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.993973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.993633    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.993973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.993714    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.993973 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.993893    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.994375 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.994068    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.994676 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.994651    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.995004 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.994733    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.995004 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.994993    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82dc1cff-278b-4772-b4ef-661fc700bce7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.996036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.996016    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:21.996586 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:21.996565    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:22.019813 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:22.019786    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"]
Oct 30 19:32:22.049932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:22.049900    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82dc1cff-278b-4772-b4ef-661fc700bce7-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:22.050080 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:22.050024    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz9fq\" (UniqueName: \"kubernetes.io/projected/82dc1cff-278b-4772-b4ef-661fc700bce7-kube-api-access-kz9fq\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-4wgl6\" (UID: \"82dc1cff-278b-4772-b4ef-661fc700bce7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:22.134065 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:22.134040    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:22.567615 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:22.567510    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"]
Oct 30 19:32:22.575039 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:32:22.575002    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82dc1cff_278b_4772_b4ef_661fc700bce7.slice/crio-b2e16267bd43d30931ae7e92b3627875790255fb01697e7bc54d82139bdda933 WatchSource:0}: Error finding container b2e16267bd43d30931ae7e92b3627875790255fb01697e7bc54d82139bdda933: Status 404 returned error can't find the container with id b2e16267bd43d30931ae7e92b3627875790255fb01697e7bc54d82139bdda933
Oct 30 19:32:22.577543 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:22.577498    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:32:22.577653 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:22.577602    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:32:22.577719 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:22.577650    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:32:22.949886 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:22.949845    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6" event={"ID":"82dc1cff-278b-4772-b4ef-661fc700bce7","Type":"ContainerStarted","Data":"26299304ac96d42b3fd9109992a59c9cb7b9bb6276a8efb444d412c71082500e"}
Oct 30 19:32:22.950083 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:22.949916    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6" event={"ID":"82dc1cff-278b-4772-b4ef-661fc700bce7","Type":"ContainerStarted","Data":"b2e16267bd43d30931ae7e92b3627875790255fb01697e7bc54d82139bdda933"}
Oct 30 19:32:23.112683 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:23.112611    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6" podStartSLOduration=2.112595727 podStartE2EDuration="2.112595727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:23.11171102 +0000 UTC m=+3249.282067870" watchObservedRunningTime="2026-10-30 19:32:23.112595727 +0000 UTC m=+3249.282952574"
Oct 30 19:32:23.137552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:23.136880    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:23.141564 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:23.138801    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:32:23.141564 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:23.138859    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6" podUID="82dc1cff-278b-4772-b4ef-661fc700bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:32:24.137552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:24.136859    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:32:24.137552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:24.136920    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6" podUID="82dc1cff-278b-4772-b4ef-661fc700bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:32:25.137353 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:25.136904    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:32:25.137353 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:25.136965    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6" podUID="82dc1cff-278b-4772-b4ef-661fc700bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:32:25.844060 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:25.844015    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 30 19:32:26.137451 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:26.137355    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:32:26.137732 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:26.137702    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6" podUID="82dc1cff-278b-4772-b4ef-661fc700bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:32:27.137554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:27.136844    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:32:27.137554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:27.136905    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6" podUID="82dc1cff-278b-4772-b4ef-661fc700bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:32:28.140556 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:28.136888    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:32:28.140556 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:28.136949    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6" podUID="82dc1cff-278b-4772-b4ef-661fc700bce7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 19:32:29.138993 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:29.138958    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:29.139388 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:29.139369    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:29.140049 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:29.140033    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-4wgl6"
Oct 30 19:32:30.946754 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:30.946709    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Oct 30 19:32:35.843686 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:35.843638    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 30 19:32:36.926890 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:36.926858    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"]
Oct 30 19:32:36.934636 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:36.934041    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.023626 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.023597    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"]
Oct 30 19:32:37.051483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.051442    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.051658 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.051562    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.051658 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.051619    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7rp\" (UniqueName: \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-kube-api-access-xr7rp\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.051658 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.051646    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.051804 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.051670    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.051804 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.051696    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.051804 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.051759    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.051804 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.051795    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.051925 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.051869    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153091 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153056    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153091 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153095    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153321 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153240    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153321 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153308    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153341    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153376    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7rp\" (UniqueName: \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-kube-api-access-xr7rp\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153401    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153427    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153690 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153452    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153690 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153581    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153690 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153631    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153840 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153760    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.153943 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.153919    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.154224 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.154203    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.155855 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.155831    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.156218 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.156200    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.208858 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.208794    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.209105 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.209083    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7rp\" (UniqueName: \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-kube-api-access-xr7rp\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-vxv28\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.249550 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.249497    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:37.684786 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.684745    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"]
Oct 30 19:32:37.692023 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:32:37.691987    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1ccf1b_7c4d_4b13_9918_0b65bcdebfd9.slice/crio-86fcb5ad5dc30b4a541a8a1dee1c00d5d50696fe9aaa222e6d9c122c2b18c613 WatchSource:0}: Error finding container 86fcb5ad5dc30b4a541a8a1dee1c00d5d50696fe9aaa222e6d9c122c2b18c613: Status 404 returned error can't find the container with id 86fcb5ad5dc30b4a541a8a1dee1c00d5d50696fe9aaa222e6d9c122c2b18c613
Oct 30 19:32:37.694797 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.694756    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:32:37.694880 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.694851    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:32:37.694918 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:37.694893    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:32:38.051541 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:38.051432    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" event={"ID":"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9","Type":"ContainerStarted","Data":"f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11"}
Oct 30 19:32:38.051541 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:38.051503    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" event={"ID":"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9","Type":"ContainerStarted","Data":"86fcb5ad5dc30b4a541a8a1dee1c00d5d50696fe9aaa222e6d9c122c2b18c613"}
Oct 30 19:32:38.168880 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:38.168821    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" podStartSLOduration=2.168802354 podStartE2EDuration="2.168802354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:38.164271399 +0000 UTC m=+3264.334628248" watchObservedRunningTime="2026-10-30 19:32:38.168802354 +0000 UTC m=+3264.339159203"
Oct 30 19:32:38.253555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:38.253277    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:38.253555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:38.253427    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-vxv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:32:38.253555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:38.253475    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:32:39.253554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:39.252912    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-vxv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:32:39.253554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:39.252977    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:32:40.249922 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:40.249884    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-vxv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:32:40.250204 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:40.250172    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:32:40.950676 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:40.950632    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Oct 30 19:32:41.253611 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:41.252892    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-vxv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:32:41.253611 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:41.252954    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:32:42.252556 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:42.251986    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-vxv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:32:42.252556 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:42.252072    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:32:43.251555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:43.249921    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-vxv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:32:43.251555 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:43.250004    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 19:32:44.254912 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:44.254877    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:44.255396 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:44.255175    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:44.255987 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:44.255967    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:32:45.843391 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:45.843348    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 30 19:32:49.873202 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:32:49.873162    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:32:50.946424 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:50.946372    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Oct 30 19:32:55.843372 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:32:55.843316    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 30 19:33:00.964513 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:00.964465    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:33:00.977914 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:00.977879    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:33:05.852657 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:05.852623    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:33:05.860291 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:05.860260    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:33:07.046483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:07.046445    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:13.549631 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:13.549584    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"]
Oct 30 19:33:13.599984 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:13.599955    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:13.813396 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:13.813314    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:19.873553 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:33:19.873486    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:33:28.577760 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:28.577716    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main" containerID="cri-o://aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa" gracePeriod=45
Oct 30 19:33:29.551628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.551602    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:33:29.619753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.619696    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-model-cache\") pod \"660f0ed2-bac3-4b78-90ee-b0546b949747\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") "
Oct 30 19:33:29.619753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.619751    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660f0ed2-bac3-4b78-90ee-b0546b949747-tls-certs\" (UniqueName: \"kubernetes.io/secret/660f0ed2-bac3-4b78-90ee-b0546b949747-tls-certs\") pod \"660f0ed2-bac3-4b78-90ee-b0546b949747\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") "
Oct 30 19:33:29.620206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.619769    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-dshm\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-dshm\") pod \"660f0ed2-bac3-4b78-90ee-b0546b949747\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") "
Oct 30 19:33:29.620206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.619794    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660f0ed2-bac3-4b78-90ee-b0546b949747-kube-api-access-54h87\" (UniqueName: \"kubernetes.io/projected/660f0ed2-bac3-4b78-90ee-b0546b949747-kube-api-access-54h87\") pod \"660f0ed2-bac3-4b78-90ee-b0546b949747\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") "
Oct 30 19:33:29.620206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.619844    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-kserve-provision-location\") pod \"660f0ed2-bac3-4b78-90ee-b0546b949747\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") "
Oct 30 19:33:29.620206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.619873    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-tmp-dir\") pod \"660f0ed2-bac3-4b78-90ee-b0546b949747\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") "
Oct 30 19:33:29.620206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.619903    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-home\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-home\") pod \"660f0ed2-bac3-4b78-90ee-b0546b949747\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") "
Oct 30 19:33:29.620206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.619936    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660f0ed2-bac3-4b78-90ee-b0546b949747-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660f0ed2-bac3-4b78-90ee-b0546b949747-cabundle-cert\") pod \"660f0ed2-bac3-4b78-90ee-b0546b949747\" (UID: \"660f0ed2-bac3-4b78-90ee-b0546b949747\") "
Oct 30 19:33:29.620206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.620024    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-model-cache" pod "660f0ed2-bac3-4b78-90ee-b0546b949747" (UID: "660f0ed2-bac3-4b78-90ee-b0546b949747"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:29.620594 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.620230    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.620654 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.620601    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660f0ed2-bac3-4b78-90ee-b0546b949747-cabundle-cert" pod "660f0ed2-bac3-4b78-90ee-b0546b949747" (UID: "660f0ed2-bac3-4b78-90ee-b0546b949747"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:29.620801 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.620755    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-home" pod "660f0ed2-bac3-4b78-90ee-b0546b949747" (UID: "660f0ed2-bac3-4b78-90ee-b0546b949747"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:29.622776 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.622749    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660f0ed2-bac3-4b78-90ee-b0546b949747-tls-certs" pod "660f0ed2-bac3-4b78-90ee-b0546b949747" (UID: "660f0ed2-bac3-4b78-90ee-b0546b949747"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:29.623010 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.622986    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-dshm" pod "660f0ed2-bac3-4b78-90ee-b0546b949747" (UID: "660f0ed2-bac3-4b78-90ee-b0546b949747"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:29.623010 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.622998    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660f0ed2-bac3-4b78-90ee-b0546b949747-kube-api-access-54h87" pod "660f0ed2-bac3-4b78-90ee-b0546b949747" (UID: "660f0ed2-bac3-4b78-90ee-b0546b949747"). InnerVolumeSpecName "kube-api-access-54h87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:29.632717 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.632694    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-tmp-dir" pod "660f0ed2-bac3-4b78-90ee-b0546b949747" (UID: "660f0ed2-bac3-4b78-90ee-b0546b949747"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:29.654057 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.654018    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-kserve-provision-location" pod "660f0ed2-bac3-4b78-90ee-b0546b949747" (UID: "660f0ed2-bac3-4b78-90ee-b0546b949747"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:29.721641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.721612    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660f0ed2-bac3-4b78-90ee-b0546b949747-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.721641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.721640    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.721810 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.721654    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54h87\" (UniqueName: \"kubernetes.io/projected/660f0ed2-bac3-4b78-90ee-b0546b949747-kube-api-access-54h87\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.721810 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.721669    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.721810 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.721683    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.721810 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.721694    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/660f0ed2-bac3-4b78-90ee-b0546b949747-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.721810 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:29.721706    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660f0ed2-bac3-4b78-90ee-b0546b949747-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:30.326856 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.326824    4853 generic.go:356] "Generic (PLEG): container finished" podID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerID="aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa" exitCode=0
Oct 30 19:33:30.327040 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.326884    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"
Oct 30 19:33:30.327040 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.326912    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" event={"ID":"660f0ed2-bac3-4b78-90ee-b0546b949747","Type":"ContainerDied","Data":"aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa"}
Oct 30 19:33:30.327040 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.326981    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj" event={"ID":"660f0ed2-bac3-4b78-90ee-b0546b949747","Type":"ContainerDied","Data":"1e23336669017ffde55ea945c89b4ce4e0173a810ccfbbc5e547f59a3ddf7c0c"}
Oct 30 19:33:30.327040 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.327009    4853 scope.go:122] "RemoveContainer" containerID="aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa"
Oct 30 19:33:30.337092 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.337069    4853 scope.go:122] "RemoveContainer" containerID="228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83"
Oct 30 19:33:30.349075 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.349046    4853 scope.go:122] "RemoveContainer" containerID="aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa"
Oct 30 19:33:30.349682 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:33:30.349464    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa\": container with ID starting with aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa not found: ID does not exist" containerID="aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa"
Oct 30 19:33:30.349682 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.349513    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa"} err="failed to get container status \"aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa\": rpc error: code = NotFound desc = could not find container \"aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa\": container with ID starting with aae7d2d41768adebb59641072ff5486c721c889e9080b9abf767047474f10cfa not found: ID does not exist"
Oct 30 19:33:30.349682 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.349565    4853 scope.go:122] "RemoveContainer" containerID="228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83"
Oct 30 19:33:30.349978 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:33:30.349947    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83\": container with ID starting with 228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83 not found: ID does not exist" containerID="228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83"
Oct 30 19:33:30.350058 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.349991    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83"} err="failed to get container status \"228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83\": rpc error: code = NotFound desc = could not find container \"228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83\": container with ID starting with 228475c436a38d1176eca1973f90aa7a4ea719e1cfd959e5c21757588cd5cf83 not found: ID does not exist"
Oct 30 19:33:30.533644 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.533610    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"]
Oct 30 19:33:30.581643 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:30.581554    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7bfc4dd5bf-7wchj"]
Oct 30 19:33:31.776220 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:31.776188    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r_b747f6b5-a02f-4c94-a97c-a215dc31e631/storage-initializer/0.log"
Oct 30 19:33:31.896099 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:31.896066    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r_b747f6b5-a02f-4c94-a97c-a215dc31e631/main/0.log"
Oct 30 19:33:31.939779 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:31.939751    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" path="/var/lib/kubelet/pods/660f0ed2-bac3-4b78-90ee-b0546b949747/volumes"
Oct 30 19:33:36.303184 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:36.303150    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"]
Oct 30 19:33:36.375862 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:36.375831    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:46.803925 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:46.803884    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:49.874121 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:33:49.874066    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:33:51.325909 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:51.325841    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main" containerID="cri-o://e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3" gracePeriod=45
Oct 30 19:33:51.853852 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:51.853818    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 30 19:33:51.985884 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:51.985854    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:33:52.098014 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.097986    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-home\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-home\") pod \"b747f6b5-a02f-4c94-a97c-a215dc31e631\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") "
Oct 30 19:33:52.098152 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098025    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b747f6b5-a02f-4c94-a97c-a215dc31e631-kube-api-access-mksn9\" (UniqueName: \"kubernetes.io/projected/b747f6b5-a02f-4c94-a97c-a215dc31e631-kube-api-access-mksn9\") pod \"b747f6b5-a02f-4c94-a97c-a215dc31e631\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") "
Oct 30 19:33:52.098152 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098055    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b747f6b5-a02f-4c94-a97c-a215dc31e631-tls-certs\" (UniqueName: \"kubernetes.io/secret/b747f6b5-a02f-4c94-a97c-a215dc31e631-tls-certs\") pod \"b747f6b5-a02f-4c94-a97c-a215dc31e631\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") "
Oct 30 19:33:52.098152 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098109    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b747f6b5-a02f-4c94-a97c-a215dc31e631-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b747f6b5-a02f-4c94-a97c-a215dc31e631-cabundle-cert\") pod \"b747f6b5-a02f-4c94-a97c-a215dc31e631\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") "
Oct 30 19:33:52.098328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098218    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") pod \"b747f6b5-a02f-4c94-a97c-a215dc31e631\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") "
Oct 30 19:33:52.098328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098252    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-tmp-dir\") pod \"b747f6b5-a02f-4c94-a97c-a215dc31e631\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") "
Oct 30 19:33:52.098328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098321    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-dshm\") pod \"b747f6b5-a02f-4c94-a97c-a215dc31e631\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") "
Oct 30 19:33:52.098487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098349    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-kserve-provision-location\") pod \"b747f6b5-a02f-4c94-a97c-a215dc31e631\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") "
Oct 30 19:33:52.098487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098406    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-model-cache\") pod \"b747f6b5-a02f-4c94-a97c-a215dc31e631\" (UID: \"b747f6b5-a02f-4c94-a97c-a215dc31e631\") "
Oct 30 19:33:52.098487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098423    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-home" pod "b747f6b5-a02f-4c94-a97c-a215dc31e631" (UID: "b747f6b5-a02f-4c94-a97c-a215dc31e631"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:52.098677 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098488    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b747f6b5-a02f-4c94-a97c-a215dc31e631-cabundle-cert" pod "b747f6b5-a02f-4c94-a97c-a215dc31e631" (UID: "b747f6b5-a02f-4c94-a97c-a215dc31e631"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:52.098677 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098650    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:52.098677 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098667    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b747f6b5-a02f-4c94-a97c-a215dc31e631-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:52.098907 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.098884    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-model-cache" pod "b747f6b5-a02f-4c94-a97c-a215dc31e631" (UID: "b747f6b5-a02f-4c94-a97c-a215dc31e631"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:52.100880 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.100856    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b747f6b5-a02f-4c94-a97c-a215dc31e631-kube-api-access-mksn9" pod "b747f6b5-a02f-4c94-a97c-a215dc31e631" (UID: "b747f6b5-a02f-4c94-a97c-a215dc31e631"). InnerVolumeSpecName "kube-api-access-mksn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:52.101000 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.100924    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-dshm" pod "b747f6b5-a02f-4c94-a97c-a215dc31e631" (UID: "b747f6b5-a02f-4c94-a97c-a215dc31e631"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:52.101057 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.101015    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b747f6b5-a02f-4c94-a97c-a215dc31e631-tls-certs" pod "b747f6b5-a02f-4c94-a97c-a215dc31e631" (UID: "b747f6b5-a02f-4c94-a97c-a215dc31e631"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:52.103895 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.103875    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d" pod "b747f6b5-a02f-4c94-a97c-a215dc31e631" (UID: "b747f6b5-a02f-4c94-a97c-a215dc31e631"). InnerVolumeSpecName "pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 19:33:52.112698 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.112675    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-tmp-dir" pod "b747f6b5-a02f-4c94-a97c-a215dc31e631" (UID: "b747f6b5-a02f-4c94-a97c-a215dc31e631"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:52.123199 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.123178    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-kserve-provision-location" pod "b747f6b5-a02f-4c94-a97c-a215dc31e631" (UID: "b747f6b5-a02f-4c94-a97c-a215dc31e631"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:52.199024 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.198997    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:52.199119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.199027    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:52.199119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.199039    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:52.199119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.199049    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mksn9\" (UniqueName: \"kubernetes.io/projected/b747f6b5-a02f-4c94-a97c-a215dc31e631-kube-api-access-mksn9\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:52.199119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.199058    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b747f6b5-a02f-4c94-a97c-a215dc31e631-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:52.199119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.199083    4853 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" "
Oct 30 19:33:52.199119 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.199093    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b747f6b5-a02f-4c94-a97c-a215dc31e631-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:52.208759 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.208740    4853 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d") on node "ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 19:33:52.299435 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.299414    4853 reconciler_common.go:299] "Volume detached for volume \"pvc-cb441312-8a03-4f0a-a650-5f6d56d3d5b9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fe7621399299506d\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"csi-629954111aa2ac0ba579bcd4565a19ffaf5549c615d1b4d7092ec334191f49c9\""
Oct 30 19:33:52.435837 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.435768    4853 generic.go:356] "Generic (PLEG): container finished" podID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerID="e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3" exitCode=0
Oct 30 19:33:52.435837 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.435803    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" event={"ID":"b747f6b5-a02f-4c94-a97c-a215dc31e631","Type":"ContainerDied","Data":"e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3"}
Oct 30 19:33:52.436323 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.435845    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"
Oct 30 19:33:52.436323 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.435869    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r" event={"ID":"b747f6b5-a02f-4c94-a97c-a215dc31e631","Type":"ContainerDied","Data":"398172d349a93562363d34003c82a17ca1cbec2b10fdc82c9a35f5a59ede57b3"}
Oct 30 19:33:52.436323 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.435895    4853 scope.go:122] "RemoveContainer" containerID="e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3"
Oct 30 19:33:52.445454 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.445436    4853 scope.go:122] "RemoveContainer" containerID="2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6"
Oct 30 19:33:52.453866 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.453848    4853 scope.go:122] "RemoveContainer" containerID="e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3"
Oct 30 19:33:52.454127 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:33:52.454111    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3\": container with ID starting with e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3 not found: ID does not exist" containerID="e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3"
Oct 30 19:33:52.454173 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.454133    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3"} err="failed to get container status \"e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3\": rpc error: code = NotFound desc = could not find container \"e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3\": container with ID starting with e9cd0e42ff1de499f83581d47898cebc4b4f5de676dc21be7be8caa8c0ab19d3 not found: ID does not exist"
Oct 30 19:33:52.454173 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.454149    4853 scope.go:122] "RemoveContainer" containerID="2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6"
Oct 30 19:33:52.454384 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:33:52.454363    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6\": container with ID starting with 2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6 not found: ID does not exist" containerID="2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6"
Oct 30 19:33:52.454441 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.454394    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6"} err="failed to get container status \"2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6\": rpc error: code = NotFound desc = could not find container \"2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6\": container with ID starting with 2b5c35729b23c903589ac831310d45e2797be1bf1b7900b405434990dadd31d6 not found: ID does not exist"
Oct 30 19:33:52.596049 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.596027    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"]
Oct 30 19:33:52.642670 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:52.642638    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6bddd54cf8-9p74r"]
Oct 30 19:33:53.939210 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:53.939167    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" path="/var/lib/kubelet/pods/b747f6b5-a02f-4c94-a97c-a215dc31e631/volumes"
Oct 30 19:33:56.768076 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:56.768034    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:56.984856 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:56.984823    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:57.197463 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:57.197426    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:57.433892 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:57.433866    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:57.641483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:57.641456    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:59.772838 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:59.772796    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:59.977280 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:33:59.977246    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:00.328556 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.328501    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"]
Oct 30 19:34:00.328877 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.328861    4853 cpu_manager.go:414] "containerMap: removing container" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main"
Oct 30 19:34:00.328929 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.328878    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main"
Oct 30 19:34:00.328929 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.328890    4853 cpu_manager.go:414] "containerMap: removing container" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main"
Oct 30 19:34:00.328929 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.328897    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main"
Oct 30 19:34:00.328929 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.328914    4853 cpu_manager.go:414] "containerMap: removing container" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="storage-initializer"
Oct 30 19:34:00.328929 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.328921    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="storage-initializer"
Oct 30 19:34:00.328929 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.328928    4853 cpu_manager.go:414] "containerMap: removing container" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="storage-initializer"
Oct 30 19:34:00.328929 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.328933    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="storage-initializer"
Oct 30 19:34:00.329141 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.329007    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="660f0ed2-bac3-4b78-90ee-b0546b949747" containerName="main"
Oct 30 19:34:00.329141 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.329014    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="b747f6b5-a02f-4c94-a97c-a215dc31e631" containerName="main"
Oct 30 19:34:00.333980 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.333957    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:00.385257 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.385225    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"]
Oct 30 19:34:00.430124 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.430096    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Oct 30 19:34:00.430247 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.430145    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Oct 30 19:34:00.430247 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.430146    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Oct 30 19:34:00.430656 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.430638    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-mplg4\""
Oct 30 19:34:00.523383 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.523359    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:00.562089 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.562055    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbf2\" (UniqueName: \"kubernetes.io/projected/6fc653ca-6ead-4e46-b264-cf3afbbd0c17-kube-api-access-vlbf2\") pod \"e2e-pvc-lora-download-dlg9q\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:00.663027 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.662966    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbf2\" (UniqueName: \"kubernetes.io/projected/6fc653ca-6ead-4e46-b264-cf3afbbd0c17-kube-api-access-vlbf2\") pod \"e2e-pvc-lora-download-dlg9q\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:00.721549 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:00.721497    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbf2\" (UniqueName: \"kubernetes.io/projected/6fc653ca-6ead-4e46-b264-cf3afbbd0c17-kube-api-access-vlbf2\") pod \"e2e-pvc-lora-download-dlg9q\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:01.667031 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:01.666994    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"]
Oct 30 19:34:01.672968 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:01.672936    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-ghl67"]
Oct 30 19:34:01.865357 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:01.865336    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:01.938672 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:01.938613    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9f8e0f-67c5-4614-b210-b9d3ff3306b4" path="/var/lib/kubelet/pods/dd9f8e0f-67c5-4614-b210-b9d3ff3306b4/volumes"
Oct 30 19:34:02.071515 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:02.071488    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:02.281590 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:02.281509    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:03.133573 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:03.133514    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:03.342307 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:03.342270    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:04.416826 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:04.416795    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:07.421253 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:07.421216    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:08.167760 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.167720    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:08.799338 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.799305    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"]
Oct 30 19:34:08.803079 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.803057    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:08.819766 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.819742    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"e2e-pvc-lora-download-dlg9q\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:08.851421 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.851389    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"]
Oct 30 19:34:08.891353 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.891328    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Oct 30 19:34:08.891353 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.891328    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:34:08.891576 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.891340    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Oct 30 19:34:08.891576 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.891388    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-px5hr\""
Oct 30 19:34:08.891576 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.891392    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Oct 30 19:34:08.921054 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.921023    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:08.921155 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.921065    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61118ca8-7426-4899-a770-4fd2ef42fede-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:08.921207 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.921177    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:08.921243 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.921206    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61118ca8-7426-4899-a770-4fd2ef42fede-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:08.921313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.921296    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:08.921361 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.921345    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:08.921408 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.921362    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:08.921408 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.921380    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8klv\" (UniqueName: \"kubernetes.io/projected/61118ca8-7426-4899-a770-4fd2ef42fede-kube-api-access-b8klv\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:08.932304 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.932275    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:08.943812 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.943788    4853 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"e2e-pvc-lora-download-dlg9q\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:08.955090 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:08.955071    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Oct 30 19:34:09.022283 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022263    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.022401 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022305    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"e2e-pvc-lora-download-dlg9q\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:09.022401 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022323    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.022401 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022339    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.022401 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022358    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8klv\" (UniqueName: \"kubernetes.io/projected/61118ca8-7426-4899-a770-4fd2ef42fede-kube-api-access-b8klv\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.022401 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022387    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.022696 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022411    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61118ca8-7426-4899-a770-4fd2ef42fede-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.022696 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022413    4853 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"e2e-pvc-lora-download-dlg9q\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:09.022696 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022453    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.022696 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022482    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61118ca8-7426-4899-a770-4fd2ef42fede-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.022898 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022778    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.022898 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022887    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.023034 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022941    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.023034 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.022993    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.023188 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.023169    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61118ca8-7426-4899-a770-4fd2ef42fede-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.024988 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.024970    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.025248 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.025229    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61118ca8-7426-4899-a770-4fd2ef42fede-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.071897 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.071843    4853 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"e2e-pvc-lora-download-dlg9q\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") DevicePath \"csi-49511a10e62e874a2871bcde94ded5861c4b0c283397351d93b92d89fda25591\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:09.078453 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.078425    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8klv\" (UniqueName: \"kubernetes.io/projected/61118ca8-7426-4899-a770-4fd2ef42fede-kube-api-access-b8klv\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.119398 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.119359    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:09.418589 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.418563    4853 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"e2e-pvc-lora-download-dlg9q\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/85add36262e3a2c49235e99a6fa627f9b94bde9afc08b5a7e626acc11087f42d/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:09.477927 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.477904    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"e2e-pvc-lora-download-dlg9q\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:09.599758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.599724    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"]
Oct 30 19:34:09.606059 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:34:09.606035    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61118ca8_7426_4899_a770_4fd2ef42fede.slice/crio-5ca1ef12bc12be1715544d6f61d6bbb904d1ff4465dcdc199fd63ffcb75d391f WatchSource:0}: Error finding container 5ca1ef12bc12be1715544d6f61d6bbb904d1ff4465dcdc199fd63ffcb75d391f: Status 404 returned error can't find the container with id 5ca1ef12bc12be1715544d6f61d6bbb904d1ff4465dcdc199fd63ffcb75d391f
Oct 30 19:34:09.646887 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:09.646838    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:10.019733 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:10.019658    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"]
Oct 30 19:34:10.026346 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:34:10.026316    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fc653ca_6ead_4e46_b264_cf3afbbd0c17.slice/crio-d32aa58425c77d4cf6629d489ac7649abed498228e8aab7ae6bd307ae7c619b4 WatchSource:0}: Error finding container d32aa58425c77d4cf6629d489ac7649abed498228e8aab7ae6bd307ae7c619b4: Status 404 returned error can't find the container with id d32aa58425c77d4cf6629d489ac7649abed498228e8aab7ae6bd307ae7c619b4
Oct 30 19:34:10.526189 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:10.526150    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" event={"ID":"61118ca8-7426-4899-a770-4fd2ef42fede","Type":"ContainerStarted","Data":"3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41"}
Oct 30 19:34:10.526398 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:10.526216    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" event={"ID":"61118ca8-7426-4899-a770-4fd2ef42fede","Type":"ContainerStarted","Data":"5ca1ef12bc12be1715544d6f61d6bbb904d1ff4465dcdc199fd63ffcb75d391f"}
Oct 30 19:34:10.527956 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:10.527929    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q" event={"ID":"6fc653ca-6ead-4e46-b264-cf3afbbd0c17","Type":"ContainerStarted","Data":"444109a996d06b4f400d56a12fbb54af31ba005abf35518cf824da9313792856"}
Oct 30 19:34:10.528072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:10.527963    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q" event={"ID":"6fc653ca-6ead-4e46-b264-cf3afbbd0c17","Type":"ContainerStarted","Data":"d32aa58425c77d4cf6629d489ac7649abed498228e8aab7ae6bd307ae7c619b4"}
Oct 30 19:34:10.751350 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:10.750330    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q" podStartSLOduration=13.75031139 podStartE2EDuration="13.75031139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:10.749523273 +0000 UTC m=+3356.919880122" watchObservedRunningTime="2026-10-30 19:34:10.75031139 +0000 UTC m=+3356.920668241"
Oct 30 19:34:11.087912 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:11.087871    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:11.293302 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:11.293269    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:12.541291 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:12.541253    4853 generic.go:356] "Generic (PLEG): container finished" podID="61118ca8-7426-4899-a770-4fd2ef42fede" containerID="3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41" exitCode=0
Oct 30 19:34:12.541846 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:12.541332    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" event={"ID":"61118ca8-7426-4899-a770-4fd2ef42fede","Type":"ContainerDied","Data":"3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41"}
Oct 30 19:34:13.549268 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:13.549237    4853 generic.go:356] "Generic (PLEG): container finished" podID="6fc653ca-6ead-4e46-b264-cf3afbbd0c17" containerID="444109a996d06b4f400d56a12fbb54af31ba005abf35518cf824da9313792856" exitCode=0
Oct 30 19:34:13.549662 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:13.549384    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q" event={"ID":"6fc653ca-6ead-4e46-b264-cf3afbbd0c17","Type":"ContainerDied","Data":"444109a996d06b4f400d56a12fbb54af31ba005abf35518cf824da9313792856"}
Oct 30 19:34:13.551623 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:13.551595    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" event={"ID":"61118ca8-7426-4899-a770-4fd2ef42fede","Type":"ContainerStarted","Data":"e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11"}
Oct 30 19:34:13.771333 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:13.771281    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podStartSLOduration=3.392473413 podStartE2EDuration="5.771267671s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.378794258s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:13.770620326 +0000 UTC m=+3359.940977173" watchObservedRunningTime="2026-10-30 19:34:13.771267671 +0000 UTC m=+3359.941624559"
Oct 30 19:34:14.120467 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:14.120436    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:14.688509 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:14.688483    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:14.875377 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:14.875348    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") "
Oct 30 19:34:14.875584 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:14.875442    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc653ca-6ead-4e46-b264-cf3afbbd0c17-kube-api-access-vlbf2\" (UniqueName: \"kubernetes.io/projected/6fc653ca-6ead-4e46-b264-cf3afbbd0c17-kube-api-access-vlbf2\") pod \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\" (UID: \"6fc653ca-6ead-4e46-b264-cf3afbbd0c17\") "
Oct 30 19:34:14.878574 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:14.878518    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc653ca-6ead-4e46-b264-cf3afbbd0c17-kube-api-access-vlbf2" pod "6fc653ca-6ead-4e46-b264-cf3afbbd0c17" (UID: "6fc653ca-6ead-4e46-b264-cf3afbbd0c17"). InnerVolumeSpecName "kube-api-access-vlbf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:14.878699 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:14.878582    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02" pod "6fc653ca-6ead-4e46-b264-cf3afbbd0c17" (UID: "6fc653ca-6ead-4e46-b264-cf3afbbd0c17"). InnerVolumeSpecName "pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 19:34:14.977116 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:14.977077    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlbf2\" (UniqueName: \"kubernetes.io/projected/6fc653ca-6ead-4e46-b264-cf3afbbd0c17-kube-api-access-vlbf2\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:14.977255 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:14.977128    4853 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" "
Oct 30 19:34:15.562661 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:15.562623    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q" event={"ID":"6fc653ca-6ead-4e46-b264-cf3afbbd0c17","Type":"ContainerDied","Data":"d32aa58425c77d4cf6629d489ac7649abed498228e8aab7ae6bd307ae7c619b4"}
Oct 30 19:34:15.562830 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:15.562670    4853 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d32aa58425c77d4cf6629d489ac7649abed498228e8aab7ae6bd307ae7c619b4"
Oct 30 19:34:15.562830 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:15.562675    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"
Oct 30 19:34:15.951581 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:15.951523    4853 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02") on node "ip-10-0-129-31.us-west-2.compute.internal"
Oct 30 19:34:15.984998 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:15.984975    4853 reconciler_common.go:299] "Volume detached for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"csi-49511a10e62e874a2871bcde94ded5861c4b0c283397351d93b92d89fda25591\""
Oct 30 19:34:19.120725 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:19.120694    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:34:19.122235 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:19.122207    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Oct 30 19:34:19.875436 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:34:19.875397    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:34:21.659499 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:21.659461    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:29.120300 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:29.120250    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Oct 30 19:34:30.203664 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:30.203625    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:31.723098 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:31.723061    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:38.645102 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:38.645037    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:39.120145 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:39.120109    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Oct 30 19:34:47.326717 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:47.326676    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:49.120728 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:49.120687    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Oct 30 19:34:49.872117 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:34:49.872074    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:34:54.973720 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:54.973681    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:55.186046 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:55.186010    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:59.120211 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:34:59.120172    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Oct 30 19:35:07.707582 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:35:07.707521    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:09.120678 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:35:09.120638    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Oct 30 19:35:19.120385 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:35:19.120331    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Oct 30 19:35:19.874890 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:35:19.874850    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:35:29.120365 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:35:29.120315    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Oct 30 19:35:39.120713 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:35:39.120675    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Oct 30 19:35:49.120340 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:35:49.120288    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": dial tcp 10.134.0.73:8000: connect: connection refused"
Oct 30 19:35:49.873387 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:35:49.873351    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:35:51.894496 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:35:51.894454    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:59.130601 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:35:59.130556    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:35:59.139104 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:35:59.139075    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:36:06.364190 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:06.364161    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"]
Oct 30 19:36:06.445302 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:06.445269    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:08.047694 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:08.047656    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:19.088014 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:19.087954    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:19.872682 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:36:19.872645    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:36:21.387510 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:21.387464    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main" containerID="cri-o://e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11" gracePeriod=45
Oct 30 19:36:22.129335 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.129313    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:36:22.166715 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.166657    4853 generic.go:356] "Generic (PLEG): container finished" podID="61118ca8-7426-4899-a770-4fd2ef42fede" containerID="e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11" exitCode=0
Oct 30 19:36:22.166715 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.166708    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"
Oct 30 19:36:22.166859 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.166735    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" event={"ID":"61118ca8-7426-4899-a770-4fd2ef42fede","Type":"ContainerDied","Data":"e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11"}
Oct 30 19:36:22.166859 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.166794    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc" event={"ID":"61118ca8-7426-4899-a770-4fd2ef42fede","Type":"ContainerDied","Data":"5ca1ef12bc12be1715544d6f61d6bbb904d1ff4465dcdc199fd63ffcb75d391f"}
Oct 30 19:36:22.166859 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.166823    4853 scope.go:122] "RemoveContainer" containerID="e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11"
Oct 30 19:36:22.175002 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.174988    4853 scope.go:122] "RemoveContainer" containerID="3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41"
Oct 30 19:36:22.182374 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.182351    4853 scope.go:122] "RemoveContainer" containerID="e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11"
Oct 30 19:36:22.182685 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:36:22.182653    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11\": container with ID starting with e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11 not found: ID does not exist" containerID="e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11"
Oct 30 19:36:22.182781 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.182680    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11"} err="failed to get container status \"e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11\": rpc error: code = NotFound desc = could not find container \"e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11\": container with ID starting with e766dbd050b6d4578de604ed23553d009b6f24d35fd7cc733a5f882c3d594d11 not found: ID does not exist"
Oct 30 19:36:22.182781 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.182699    4853 scope.go:122] "RemoveContainer" containerID="3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41"
Oct 30 19:36:22.182925 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:36:22.182909    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41\": container with ID starting with 3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41 not found: ID does not exist" containerID="3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41"
Oct 30 19:36:22.182964 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.182928    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41"} err="failed to get container status \"3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41\": rpc error: code = NotFound desc = could not find container \"3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41\": container with ID starting with 3f6f972af87e85eeb58170c7500cf8cc22609b0150429f500623f0707f92aa41 not found: ID does not exist"
Oct 30 19:36:22.207320 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.207295    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-home\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-home\") pod \"61118ca8-7426-4899-a770-4fd2ef42fede\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") "
Oct 30 19:36:22.207398 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.207335    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61118ca8-7426-4899-a770-4fd2ef42fede-tls-certs\" (UniqueName: \"kubernetes.io/secret/61118ca8-7426-4899-a770-4fd2ef42fede-tls-certs\") pod \"61118ca8-7426-4899-a770-4fd2ef42fede\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") "
Oct 30 19:36:22.207398 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.207359    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-dshm\") pod \"61118ca8-7426-4899-a770-4fd2ef42fede\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") "
Oct 30 19:36:22.207398 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.207391    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-model-cache\") pod \"61118ca8-7426-4899-a770-4fd2ef42fede\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") "
Oct 30 19:36:22.207522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.207433    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61118ca8-7426-4899-a770-4fd2ef42fede-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61118ca8-7426-4899-a770-4fd2ef42fede-cabundle-cert\") pod \"61118ca8-7426-4899-a770-4fd2ef42fede\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") "
Oct 30 19:36:22.207522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.207462    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-tmp-dir\") pod \"61118ca8-7426-4899-a770-4fd2ef42fede\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") "
Oct 30 19:36:22.207522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.207518    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-kserve-provision-location\") pod \"61118ca8-7426-4899-a770-4fd2ef42fede\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") "
Oct 30 19:36:22.207711 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.207594    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61118ca8-7426-4899-a770-4fd2ef42fede-kube-api-access-b8klv\" (UniqueName: \"kubernetes.io/projected/61118ca8-7426-4899-a770-4fd2ef42fede-kube-api-access-b8klv\") pod \"61118ca8-7426-4899-a770-4fd2ef42fede\" (UID: \"61118ca8-7426-4899-a770-4fd2ef42fede\") "
Oct 30 19:36:22.207809 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.207785    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-model-cache" pod "61118ca8-7426-4899-a770-4fd2ef42fede" (UID: "61118ca8-7426-4899-a770-4fd2ef42fede"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:22.207958 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.207934    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61118ca8-7426-4899-a770-4fd2ef42fede-cabundle-cert" pod "61118ca8-7426-4899-a770-4fd2ef42fede" (UID: "61118ca8-7426-4899-a770-4fd2ef42fede"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:22.208126 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.208103    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-home" pod "61118ca8-7426-4899-a770-4fd2ef42fede" (UID: "61118ca8-7426-4899-a770-4fd2ef42fede"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:22.210009 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.209989    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61118ca8-7426-4899-a770-4fd2ef42fede-kube-api-access-b8klv" pod "61118ca8-7426-4899-a770-4fd2ef42fede" (UID: "61118ca8-7426-4899-a770-4fd2ef42fede"). InnerVolumeSpecName "kube-api-access-b8klv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:22.210158 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.210134    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61118ca8-7426-4899-a770-4fd2ef42fede-tls-certs" pod "61118ca8-7426-4899-a770-4fd2ef42fede" (UID: "61118ca8-7426-4899-a770-4fd2ef42fede"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:22.210158 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.210152    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-dshm" pod "61118ca8-7426-4899-a770-4fd2ef42fede" (UID: "61118ca8-7426-4899-a770-4fd2ef42fede"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:22.219073 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.219047    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-tmp-dir" pod "61118ca8-7426-4899-a770-4fd2ef42fede" (UID: "61118ca8-7426-4899-a770-4fd2ef42fede"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:22.235651 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.235627    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-kserve-provision-location" pod "61118ca8-7426-4899-a770-4fd2ef42fede" (UID: "61118ca8-7426-4899-a770-4fd2ef42fede"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:22.308998 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.308976    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8klv\" (UniqueName: \"kubernetes.io/projected/61118ca8-7426-4899-a770-4fd2ef42fede-kube-api-access-b8klv\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:22.308998 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.308998    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:22.309130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.309008    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61118ca8-7426-4899-a770-4fd2ef42fede-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:22.309130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.309017    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:22.309130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.309024    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:22.309130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.309032    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61118ca8-7426-4899-a770-4fd2ef42fede-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:22.309130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.309041    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:22.309130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.309049    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61118ca8-7426-4899-a770-4fd2ef42fede-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:22.584073 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.584048    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"]
Oct 30 19:36:22.591744 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:22.591719    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f8854fdd6rqjzc"]
Oct 30 19:36:23.938598 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:23.938566    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" path="/var/lib/kubelet/pods/61118ca8-7426-4899-a770-4fd2ef42fede/volumes"
Oct 30 19:36:25.841216 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:25.841180    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:26.049205 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:26.049160    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:32.829003 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:32.828963    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:39.404035 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.403996    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:39.616174 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.616144    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:39.818610 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.818512    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:39.914686 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.914649    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"]
Oct 30 19:36:39.914967 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.914954    4853 cpu_manager.go:414] "containerMap: removing container" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="storage-initializer"
Oct 30 19:36:39.915012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.914968    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="storage-initializer"
Oct 30 19:36:39.915012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.914980    4853 cpu_manager.go:414] "containerMap: removing container" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main"
Oct 30 19:36:39.915012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.914985    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main"
Oct 30 19:36:39.915012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.915002    4853 cpu_manager.go:414] "containerMap: removing container" podUID="6fc653ca-6ead-4e46-b264-cf3afbbd0c17" containerName="storage-initializer"
Oct 30 19:36:39.915012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.915007    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc653ca-6ead-4e46-b264-cf3afbbd0c17" containerName="storage-initializer"
Oct 30 19:36:39.915173 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.915063    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="61118ca8-7426-4899-a770-4fd2ef42fede" containerName="main"
Oct 30 19:36:39.915173 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.915073    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc653ca-6ead-4e46-b264-cf3afbbd0c17" containerName="storage-initializer"
Oct 30 19:36:39.919339 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.919319    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:39.994490 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.994455    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"kube-root-ca.crt\""
Oct 30 19:36:39.994490 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.994470    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"model-routing-only-gateway-openshift-default-dockercfg-xpmgl\""
Oct 30 19:36:39.994490 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:39.994482    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"openshift-service-ca.crt\""
Oct 30 19:36:40.016329 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.016305    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:36:40.018239 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.018219    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"]
Oct 30 19:36:40.031716 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.031692    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5788ec-1675-4664-8273-8800c084b91b-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.031841 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.031727    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.031841 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.031755    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.031841 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.031816    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgp7k\" (UniqueName: \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-kube-api-access-qgp7k\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.032001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.031881    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-data\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.032038 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.032002    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.032072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.032037    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5788ec-1675-4664-8273-8800c084b91b-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.032127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.032103    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.032170 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.032155    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-istio-token\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.070833 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.070748    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:40.133484 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.133456    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.133662 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.133492    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5788ec-1675-4664-8273-8800c084b91b-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.133662 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.133512    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.133662 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.133544    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-istio-token\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.133662 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.133608    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5788ec-1675-4664-8273-8800c084b91b-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.133662 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.133639    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.133931 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.133678    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.133931 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.133795    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgp7k\" (UniqueName: \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-kube-api-access-qgp7k\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.133931 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.133839    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-data\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.134088 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.134038    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.134155 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.134094    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.134155 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.134134    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-data\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.134265 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.134167    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.134401 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.134331    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5788ec-1675-4664-8273-8800c084b91b-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.136156 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.136136    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.136428 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.136407    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5788ec-1675-4664-8273-8800c084b91b-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.190549 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.190505    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-istio-token\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.194667 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.194644    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgp7k\" (UniqueName: \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-kube-api-access-qgp7k\") pod \"model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.234798 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.234774    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:40.270567 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.270541    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:40.478946 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.478914    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:40.658599 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.658569    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"]
Oct 30 19:36:40.665049 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:36:40.665023    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5788ec_1675_4664_8273_8800c084b91b.slice/crio-b324dbf6abba1a7a8e349f3fd78253aa4060d7600fd2ff04ea8cb6efd434861c WatchSource:0}: Error finding container b324dbf6abba1a7a8e349f3fd78253aa4060d7600fd2ff04ea8cb6efd434861c: Status 404 returned error can't find the container with id b324dbf6abba1a7a8e349f3fd78253aa4060d7600fd2ff04ea8cb6efd434861c
Oct 30 19:36:40.667023 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.666992    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:36:40.667086 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.667056    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:36:40.667136 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:40.667084    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:36:41.075332 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.075303    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:41.256388 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.256356    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" event={"ID":"1c5788ec-1675-4664-8273-8800c084b91b","Type":"ContainerStarted","Data":"028785bb62aa232e1a73cb66b3b0387766b1136137cb851687a808650d50fa87"}
Oct 30 19:36:41.256595 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.256403    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" event={"ID":"1c5788ec-1675-4664-8273-8800c084b91b","Type":"ContainerStarted","Data":"b324dbf6abba1a7a8e349f3fd78253aa4060d7600fd2ff04ea8cb6efd434861c"}
Oct 30 19:36:41.279515 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.279486    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:41.370046 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.370003    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" podStartSLOduration=2.36999074 podStartE2EDuration="2.36999074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:41.368250477 +0000 UTC m=+3507.538607337" watchObservedRunningTime="2026-10-30 19:36:41.36999074 +0000 UTC m=+3507.540347588"
Oct 30 19:36:41.416732 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.416710    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 19:36:41.423725 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.422326    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.444772 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.444749    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.444866 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.444777    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b750d2-b111-47c3-b3aa-152e7e59ef34-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.444866 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.444797    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.444866 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.444848    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.444971 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.444956    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60b750d2-b111-47c3-b3aa-152e7e59ef34-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.445002 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.444980    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvvzd\" (UniqueName: \"kubernetes.io/projected/60b750d2-b111-47c3-b3aa-152e7e59ef34-kube-api-access-pvvzd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.445036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.445016    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.445124 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.445112    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.523306 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.523280    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 19:36:41.564400 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.564258    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-j99zj\""
Oct 30 19:36:41.564646 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.564410    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Oct 30 19:36:41.564646 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.564471    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Oct 30 19:36:41.564646 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.564603    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Oct 30 19:36:41.564831 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.564670    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:36:41.564831 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.564779    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-xttbp\""
Oct 30 19:36:41.565009 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.564992    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Oct 30 19:36:41.565370 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.565348    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60b750d2-b111-47c3-b3aa-152e7e59ef34-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.565479 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.565391    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvvzd\" (UniqueName: \"kubernetes.io/projected/60b750d2-b111-47c3-b3aa-152e7e59ef34-kube-api-access-pvvzd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.565479 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.565436    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.565479 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.565464    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.565680 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.565515    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.565680 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.565556    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b750d2-b111-47c3-b3aa-152e7e59ef34-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.565680 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.565586    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.565680 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.565617    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.566088 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.566055    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.566729 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.566706    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60b750d2-b111-47c3-b3aa-152e7e59ef34-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.567301 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.567276    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.569849 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.567848    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.570453 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.570434    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.572450 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.572423    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:41.575556 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.575510    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b750d2-b111-47c3-b3aa-152e7e59ef34-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.575934 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.575892    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.624722 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.624659    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvvzd\" (UniqueName: \"kubernetes.io/projected/60b750d2-b111-47c3-b3aa-152e7e59ef34-kube-api-access-pvvzd\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:41.739150 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:41.739110    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:42.171035 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:42.171006    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 19:36:42.177650 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:36:42.177622    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b750d2_b111_47c3_b3aa_152e7e59ef34.slice/crio-d2eafe5c4d841182b50c33eae816c3cd9f947b2daf3f50cde935382989cf18ac WatchSource:0}: Error finding container d2eafe5c4d841182b50c33eae816c3cd9f947b2daf3f50cde935382989cf18ac: Status 404 returned error can't find the container with id d2eafe5c4d841182b50c33eae816c3cd9f947b2daf3f50cde935382989cf18ac
Oct 30 19:36:42.235206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:42.235182    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:42.240251 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:42.240229    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:42.263954 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:42.263924    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"60b750d2-b111-47c3-b3aa-152e7e59ef34","Type":"ContainerStarted","Data":"89b17fa00881bafe38c9f66043a0c413674b95df81acd09654745655e5f4476a"}
Oct 30 19:36:42.264067 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:42.263980    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"60b750d2-b111-47c3-b3aa-152e7e59ef34","Type":"ContainerStarted","Data":"d2eafe5c4d841182b50c33eae816c3cd9f947b2daf3f50cde935382989cf18ac"}
Oct 30 19:36:42.264235 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:42.264216    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:42.265082 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:42.265065    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:36:44.084835 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:44.084783    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:44.215242 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:44.215189    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"]
Oct 30 19:36:44.217665 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:44.217634    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-dlg9q"]
Oct 30 19:36:44.665033 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:44.664996    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:44.869897 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:44.869847    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:45.076651 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:45.076608    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:45.284453 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:45.284378    4853 generic.go:356] "Generic (PLEG): container finished" podID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerID="89b17fa00881bafe38c9f66043a0c413674b95df81acd09654745655e5f4476a" exitCode=0
Oct 30 19:36:45.284828 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:45.284446    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"60b750d2-b111-47c3-b3aa-152e7e59ef34","Type":"ContainerDied","Data":"89b17fa00881bafe38c9f66043a0c413674b95df81acd09654745655e5f4476a"}
Oct 30 19:36:45.951136 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:45.951091    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc653ca-6ead-4e46-b264-cf3afbbd0c17" path="/var/lib/kubelet/pods/6fc653ca-6ead-4e46-b264-cf3afbbd0c17/volumes"
Oct 30 19:36:46.298498 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:46.298417    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"60b750d2-b111-47c3-b3aa-152e7e59ef34","Type":"ContainerStarted","Data":"a5ca27342e3456b68982deb0aac8d0449c4a9d4b1d1c57fbc902f3d0400f4403"}
Oct 30 19:36:46.505625 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:46.505566    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.22588835 podStartE2EDuration="5.505551525s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.279663175s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:46.457632479 +0000 UTC m=+3512.627989327" watchObservedRunningTime="2026-10-30 19:36:46.505551525 +0000 UTC m=+3512.675908373"
Oct 30 19:36:46.739772 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:46.739737    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:46.831982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:46.831952    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:49.007922 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:49.007882    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:49.874828 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:36:49.874792    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:36:50.531057 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:50.531019    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:51.740097 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:51.740056    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:36:51.741776 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:51.741739    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 19:36:53.741964 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:53.741928    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:53.943928 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:36:53.943881    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:01.406879 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:01.406795    4853 generic.go:356] "Generic (PLEG): container finished" podID="1c5788ec-1675-4664-8273-8800c084b91b" containerID="028785bb62aa232e1a73cb66b3b0387766b1136137cb851687a808650d50fa87" exitCode=0
Oct 30 19:37:01.406879 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:01.406844    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" event={"ID":"1c5788ec-1675-4664-8273-8800c084b91b","Type":"ContainerDied","Data":"028785bb62aa232e1a73cb66b3b0387766b1136137cb851687a808650d50fa87"}
Oct 30 19:37:01.407330 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:01.407167    4853 scope.go:122] "RemoveContainer" containerID="028785bb62aa232e1a73cb66b3b0387766b1136137cb851687a808650d50fa87"
Oct 30 19:37:01.408256 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:01.408223    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:37:01.408323 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:01.408300    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:37:01.408358 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:01.408340    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:37:01.740298 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:01.740263    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 19:37:02.414441 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:02.414406    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" event={"ID":"1c5788ec-1675-4664-8273-8800c084b91b","Type":"ContainerStarted","Data":"dbb898e0838705c453c30cee797fef00bbcba50e2460b321576195e093b3a177"}
Oct 30 19:37:03.240557 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:03.237869    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:37:03.240557 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:03.238025    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:37:04.237945 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:04.237900    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:37:05.237557 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:05.236876    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:37:06.239551 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:06.236905    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:37:07.240548 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:07.237633    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 19:37:08.239176 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:08.239145    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:37:08.239559 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:08.239540    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:37:08.240152 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:08.240136    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:37:08.379220 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:08.379187    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:10.654847 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:10.654813    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:10.890131 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:10.890064    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:11.740268 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:11.740216    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 19:37:19.872358 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:37:19.872321    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:37:21.739684 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:21.739631    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 19:37:31.740265 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:31.740222    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 19:37:41.740121 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:41.740070    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 19:37:43.985359 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:43.985320    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:49.872385 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:37:49.872340    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:37:51.739914 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:37:51.739876    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 19:38:01.739690 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:01.739652    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 19:38:03.679347 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:03.679309    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:03.896199 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:03.896160    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:04.389194 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:04.389153    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:04.587591 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:04.587555    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:08.179098 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:08.179061    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:08.591466 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:08.591428    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"]
Oct 30 19:38:08.591713 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:08.591687    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy" containerID="cri-o://dbb898e0838705c453c30cee797fef00bbcba50e2460b321576195e093b3a177" gracePeriod=30
Oct 30 19:38:08.715262 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:08.715225    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:08.919005 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:08.918926    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:09.428133 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:09.428089    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"]
Oct 30 19:38:09.428670 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:09.428352    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerName="istio-proxy" containerID="cri-o://ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e" gracePeriod=30
Oct 30 19:38:09.475932 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:09.475896    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"]
Oct 30 19:38:09.476192 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:09.476158    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerName="istio-proxy" containerID="cri-o://f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11" gracePeriod=30
Oct 30 19:38:11.739857 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:11.739810    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 19:38:13.788753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.788635    4853 generic.go:356] "Generic (PLEG): container finished" podID="1c5788ec-1675-4664-8273-8800c084b91b" containerID="dbb898e0838705c453c30cee797fef00bbcba50e2460b321576195e093b3a177" exitCode=0
Oct 30 19:38:13.788753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.788693    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" event={"ID":"1c5788ec-1675-4664-8273-8800c084b91b","Type":"ContainerDied","Data":"dbb898e0838705c453c30cee797fef00bbcba50e2460b321576195e093b3a177"}
Oct 30 19:38:13.789297 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.788760    4853 scope.go:122] "RemoveContainer" containerID="028785bb62aa232e1a73cb66b3b0387766b1136137cb851687a808650d50fa87"
Oct 30 19:38:13.838649 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.838622    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:38:13.953847 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.953750    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-socket\") pod \"1c5788ec-1675-4664-8273-8800c084b91b\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") "
Oct 30 19:38:13.953847 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.953833    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-certs\") pod \"1c5788ec-1675-4664-8273-8800c084b91b\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") "
Oct 30 19:38:13.954072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.953867    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c5788ec-1675-4664-8273-8800c084b91b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5788ec-1675-4664-8273-8800c084b91b-istio-podinfo\") pod \"1c5788ec-1675-4664-8273-8800c084b91b\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") "
Oct 30 19:38:13.954072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.953891    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-credential-socket\") pod \"1c5788ec-1675-4664-8273-8800c084b91b\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") "
Oct 30 19:38:13.954072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.953921    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-envoy\") pod \"1c5788ec-1675-4664-8273-8800c084b91b\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") "
Oct 30 19:38:13.954072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.953964    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5788ec-1675-4664-8273-8800c084b91b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5788ec-1675-4664-8273-8800c084b91b-istiod-ca-cert\") pod \"1c5788ec-1675-4664-8273-8800c084b91b\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") "
Oct 30 19:38:13.954072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.953991    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-istio-token\" (UniqueName: \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-istio-token\") pod \"1c5788ec-1675-4664-8273-8800c084b91b\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") "
Oct 30 19:38:13.954072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954002    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-socket" pod "1c5788ec-1675-4664-8273-8800c084b91b" (UID: "1c5788ec-1675-4664-8273-8800c084b91b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:13.954072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954028    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-data\") pod \"1c5788ec-1675-4664-8273-8800c084b91b\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") "
Oct 30 19:38:13.954072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954067    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-kube-api-access-qgp7k\" (UniqueName: \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-kube-api-access-qgp7k\") pod \"1c5788ec-1675-4664-8273-8800c084b91b\" (UID: \"1c5788ec-1675-4664-8273-8800c084b91b\") "
Oct 30 19:38:13.954483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954167    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-certs" pod "1c5788ec-1675-4664-8273-8800c084b91b" (UID: "1c5788ec-1675-4664-8273-8800c084b91b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:13.954483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954413    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-data" pod "1c5788ec-1675-4664-8273-8800c084b91b" (UID: "1c5788ec-1675-4664-8273-8800c084b91b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:13.954483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954429    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5788ec-1675-4664-8273-8800c084b91b-istiod-ca-cert" pod "1c5788ec-1675-4664-8273-8800c084b91b" (UID: "1c5788ec-1675-4664-8273-8800c084b91b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:13.954483 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954429    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-credential-socket" pod "1c5788ec-1675-4664-8273-8800c084b91b" (UID: "1c5788ec-1675-4664-8273-8800c084b91b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:13.954821 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954805    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:13.954907 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954897    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:13.954969 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954917    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5788ec-1675-4664-8273-8800c084b91b-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:13.954969 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954931    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:13.954969 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.954945    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:13.956866 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.956831    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-envoy" pod "1c5788ec-1675-4664-8273-8800c084b91b" (UID: "1c5788ec-1675-4664-8273-8800c084b91b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:13.957478 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.957443    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-kube-api-access-qgp7k" pod "1c5788ec-1675-4664-8273-8800c084b91b" (UID: "1c5788ec-1675-4664-8273-8800c084b91b"). InnerVolumeSpecName "kube-api-access-qgp7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:13.957588 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.957501    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-istio-token" pod "1c5788ec-1675-4664-8273-8800c084b91b" (UID: "1c5788ec-1675-4664-8273-8800c084b91b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:13.958620 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:13.958599    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c5788ec-1675-4664-8273-8800c084b91b-istio-podinfo" pod "1c5788ec-1675-4664-8273-8800c084b91b" (UID: "1c5788ec-1675-4664-8273-8800c084b91b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:14.011966 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.011936    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:38:14.012120 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.012016    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:38:14.055889 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.055577    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5788ec-1675-4664-8273-8800c084b91b-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.055889 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.055610    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.055889 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.055626    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgp7k\" (UniqueName: \"kubernetes.io/projected/1c5788ec-1675-4664-8273-8800c084b91b-kube-api-access-qgp7k\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.055889 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.055639    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5788ec-1675-4664-8273-8800c084b91b-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.257763 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.257666    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-vxv28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:38:14.257763 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.257730    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:38:14.593125 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.593099    4853 scope.go:122] "RemoveContainer" containerID="dc91f6bab4f37bc2e165dc6472762786dab8a59c9f6346ad22cbc30a0e7a7aae"
Oct 30 19:38:14.750380 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.750358    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:38:14.752305 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.752288    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:38:14.794559 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.794513    4853 generic.go:356] "Generic (PLEG): container finished" podID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerID="f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11" exitCode=0
Oct 30 19:38:14.794919 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.794565    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"
Oct 30 19:38:14.794919 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.794599    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" event={"ID":"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9","Type":"ContainerDied","Data":"f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11"}
Oct 30 19:38:14.794919 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.794643    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28" event={"ID":"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9","Type":"ContainerDied","Data":"86fcb5ad5dc30b4a541a8a1dee1c00d5d50696fe9aaa222e6d9c122c2b18c613"}
Oct 30 19:38:14.794919 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.794668    4853 scope.go:122] "RemoveContainer" containerID="f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11"
Oct 30 19:38:14.795916 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.795895    4853 generic.go:356] "Generic (PLEG): container finished" podID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerID="ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e" exitCode=0
Oct 30 19:38:14.796013 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.795932    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"
Oct 30 19:38:14.796013 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.795972    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" event={"ID":"7c7bb60a-962a-401b-8cd9-413217ba989c","Type":"ContainerDied","Data":"ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e"}
Oct 30 19:38:14.796138 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.796013    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7" event={"ID":"7c7bb60a-962a-401b-8cd9-413217ba989c","Type":"ContainerDied","Data":"b811f0c0144206f4c515462ae9c87c23b0473fec8fef8796eb28d2612b2dfeff"}
Oct 30 19:38:14.797387 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.797367    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"
Oct 30 19:38:14.797477 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.797369    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9" event={"ID":"1c5788ec-1675-4664-8273-8800c084b91b","Type":"ContainerDied","Data":"b324dbf6abba1a7a8e349f3fd78253aa4060d7600fd2ff04ea8cb6efd434861c"}
Oct 30 19:38:14.803130 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.803104    4853 scope.go:122] "RemoveContainer" containerID="f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11"
Oct 30 19:38:14.803422 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:38:14.803397    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11\": container with ID starting with f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11 not found: ID does not exist" containerID="f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11"
Oct 30 19:38:14.803510 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.803424    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11"} err="failed to get container status \"f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11\": rpc error: code = NotFound desc = could not find container \"f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11\": container with ID starting with f7c159ab4d02ee19af3a34aeb1d5d5b4bfb82698bad0ad4dcb89bc0ce9a4df11 not found: ID does not exist"
Oct 30 19:38:14.803510 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.803450    4853 scope.go:122] "RemoveContainer" containerID="ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e"
Oct 30 19:38:14.811415 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.811399    4853 scope.go:122] "RemoveContainer" containerID="ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e"
Oct 30 19:38:14.811713 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:38:14.811696    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e\": container with ID starting with ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e not found: ID does not exist" containerID="ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e"
Oct 30 19:38:14.811780 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.811719    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e"} err="failed to get container status \"ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e\": rpc error: code = NotFound desc = could not find container \"ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e\": container with ID starting with ccdba1b64770da9402f5da4c70e4eae1cf4ab55a81c8664fdd2f51d2f3ccb67e not found: ID does not exist"
Oct 30 19:38:14.811780 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.811736    4853 scope.go:122] "RemoveContainer" containerID="dbb898e0838705c453c30cee797fef00bbcba50e2460b321576195e093b3a177"
Oct 30 19:38:14.861791 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.861736    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-data\") pod \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") "
Oct 30 19:38:14.861791 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.861773    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-podinfo\") pod \"7c7bb60a-962a-401b-8cd9-413217ba989c\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") "
Oct 30 19:38:14.861910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.861800    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istiod-ca-cert\") pod \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") "
Oct 30 19:38:14.861910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.861836    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-token\" (UniqueName: \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-token\") pod \"7c7bb60a-962a-401b-8cd9-413217ba989c\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") "
Oct 30 19:38:14.861910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.861856    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-socket\") pod \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") "
Oct 30 19:38:14.861910 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.861882    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-kube-api-access-2vvwl\" (UniqueName: \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-kube-api-access-2vvwl\") pod \"7c7bb60a-962a-401b-8cd9-413217ba989c\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") "
Oct 30 19:38:14.862120 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.861924    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-envoy\") pod \"7c7bb60a-962a-401b-8cd9-413217ba989c\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") "
Oct 30 19:38:14.862120 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.861953    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-certs\") pod \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") "
Oct 30 19:38:14.862120 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862012    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-socket\") pod \"7c7bb60a-962a-401b-8cd9-413217ba989c\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") "
Oct 30 19:38:14.862120 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862033    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-data" pod "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" (UID: "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:14.862120 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862044    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-data\") pod \"7c7bb60a-962a-401b-8cd9-413217ba989c\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") "
Oct 30 19:38:14.862120 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862086    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-podinfo\") pod \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") "
Oct 30 19:38:14.862120 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862117    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-token\" (UniqueName: \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-token\") pod \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") "
Oct 30 19:38:14.862482 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862159    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-credential-socket\") pod \"7c7bb60a-962a-401b-8cd9-413217ba989c\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") "
Oct 30 19:38:14.862482 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862163    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-socket" pod "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" (UID: "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:14.862482 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862186    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-envoy\") pod \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") "
Oct 30 19:38:14.862482 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862224    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-certs\") pod \"7c7bb60a-962a-401b-8cd9-413217ba989c\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") "
Oct 30 19:38:14.862482 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862264    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-kube-api-access-xr7rp\" (UniqueName: \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-kube-api-access-xr7rp\") pod \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") "
Oct 30 19:38:14.862482 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862301    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-socket" pod "7c7bb60a-962a-401b-8cd9-413217ba989c" (UID: "7c7bb60a-962a-401b-8cd9-413217ba989c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:14.862482 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862316    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istiod-ca-cert" pod "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" (UID: "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:14.862482 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862327    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-credential-socket\") pod \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\" (UID: \"1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9\") "
Oct 30 19:38:14.862482 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862376    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7bb60a-962a-401b-8cd9-413217ba989c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7bb60a-962a-401b-8cd9-413217ba989c-istiod-ca-cert\") pod \"7c7bb60a-962a-401b-8cd9-413217ba989c\" (UID: \"7c7bb60a-962a-401b-8cd9-413217ba989c\") "
Oct 30 19:38:14.862982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862487    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-certs" pod "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" (UID: "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:14.862982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862691    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-data" pod "7c7bb60a-962a-401b-8cd9-413217ba989c" (UID: "7c7bb60a-962a-401b-8cd9-413217ba989c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:14.862982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862729    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-certs" pod "7c7bb60a-962a-401b-8cd9-413217ba989c" (UID: "7c7bb60a-962a-401b-8cd9-413217ba989c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:14.862982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862831    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.862982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862852    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.862982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862868    4853 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.862982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862884    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.862982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862898    4853 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-workload-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.862982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862912    4853 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-data\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.862982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.862926    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.863509 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.863292    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-credential-socket" pod "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" (UID: "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:14.863509 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.863433    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7bb60a-962a-401b-8cd9-413217ba989c-istiod-ca-cert" pod "7c7bb60a-962a-401b-8cd9-413217ba989c" (UID: "7c7bb60a-962a-401b-8cd9-413217ba989c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:14.863815 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.863780    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-credential-socket" pod "7c7bb60a-962a-401b-8cd9-413217ba989c" (UID: "7c7bb60a-962a-401b-8cd9-413217ba989c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:14.864690 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.864667    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-envoy" pod "7c7bb60a-962a-401b-8cd9-413217ba989c" (UID: "7c7bb60a-962a-401b-8cd9-413217ba989c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:14.864960 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.864935    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-kube-api-access-2vvwl" pod "7c7bb60a-962a-401b-8cd9-413217ba989c" (UID: "7c7bb60a-962a-401b-8cd9-413217ba989c"). InnerVolumeSpecName "kube-api-access-2vvwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:14.865750 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.865716    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-token" pod "7c7bb60a-962a-401b-8cd9-413217ba989c" (UID: "7c7bb60a-962a-401b-8cd9-413217ba989c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:14.865864 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.865839    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-kube-api-access-xr7rp" pod "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" (UID: "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9"). InnerVolumeSpecName "kube-api-access-xr7rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:14.866056 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.866032    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-token" pod "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" (UID: "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:14.866140 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.866125    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-podinfo" pod "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" (UID: "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:14.866202 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.866187    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-podinfo" pod "7c7bb60a-962a-401b-8cd9-413217ba989c" (UID: "7c7bb60a-962a-401b-8cd9-413217ba989c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:14.867061 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.867034    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-envoy" pod "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" (UID: "1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:14.963312 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.963282    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.963312 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.963309    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.963512 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.963324    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.963512 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.963336    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.963512 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.963348    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr7rp\" (UniqueName: \"kubernetes.io/projected/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-kube-api-access-xr7rp\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.963512 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.963360    4853 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9-credential-socket\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.963512 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.963373    4853 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c7bb60a-962a-401b-8cd9-413217ba989c-istiod-ca-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.963512 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.963385    4853 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-podinfo\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.963512 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.963396    4853 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-token\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.963512 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.963409    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vvwl\" (UniqueName: \"kubernetes.io/projected/7c7bb60a-962a-401b-8cd9-413217ba989c-kube-api-access-2vvwl\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.963512 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:14.963422    4853 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c7bb60a-962a-401b-8cd9-413217ba989c-istio-envoy\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:15.151034 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:15.150969    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"]
Oct 30 19:38:15.156696 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:15.156674    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-7fd47b77c-7cgv9"]
Oct 30 19:38:15.264021 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:15.263998    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"]
Oct 30 19:38:15.268644 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:15.268622    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-vxv28"]
Oct 30 19:38:15.372582 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:15.372548    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"]
Oct 30 19:38:15.375857 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:15.375830    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9g8z7"]
Oct 30 19:38:15.939074 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:15.939046    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" path="/var/lib/kubelet/pods/1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9/volumes"
Oct 30 19:38:15.939458 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:15.939444    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5788ec-1675-4664-8273-8800c084b91b" path="/var/lib/kubelet/pods/1c5788ec-1675-4664-8273-8800c084b91b/volumes"
Oct 30 19:38:15.939857 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:15.939844    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" path="/var/lib/kubelet/pods/7c7bb60a-962a-401b-8cd9-413217ba989c/volumes"
Oct 30 19:38:16.705065 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:16.705023    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:19.882268 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:38:19.882228    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:38:21.749012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:21.748983    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:38:21.756435 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:21.756415    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:38:31.258747 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:31.258716    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 19:38:31.301831 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:31.301798    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:31.516581 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:31.516464    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:31.722464 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:31.722429    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:32.063681 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:32.063647    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:33.191379 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:33.191345    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:39.767574 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:39.767513    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:42.463547 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:42.463502    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:46.283868 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:46.283815    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main" containerID="cri-o://a5ca27342e3456b68982deb0aac8d0449c4a9d4b1d1c57fbc902f3d0400f4403" gracePeriod=45
Oct 30 19:38:46.749590 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:46.749523    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 19:38:46.954639 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:46.954610    4853 generic.go:356] "Generic (PLEG): container finished" podID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerID="a5ca27342e3456b68982deb0aac8d0449c4a9d4b1d1c57fbc902f3d0400f4403" exitCode=0
Oct 30 19:38:46.954787 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:46.954697    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"60b750d2-b111-47c3-b3aa-152e7e59ef34","Type":"ContainerDied","Data":"a5ca27342e3456b68982deb0aac8d0449c4a9d4b1d1c57fbc902f3d0400f4403"}
Oct 30 19:38:47.031488 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.031460    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:38:47.197888 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.197846    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60b750d2-b111-47c3-b3aa-152e7e59ef34-tls-certs\" (UniqueName: \"kubernetes.io/secret/60b750d2-b111-47c3-b3aa-152e7e59ef34-tls-certs\") pod \"60b750d2-b111-47c3-b3aa-152e7e59ef34\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") "
Oct 30 19:38:47.198036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.197918    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-model-cache\") pod \"60b750d2-b111-47c3-b3aa-152e7e59ef34\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") "
Oct 30 19:38:47.198036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.197956    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b750d2-b111-47c3-b3aa-152e7e59ef34-kube-api-access-pvvzd\" (UniqueName: \"kubernetes.io/projected/60b750d2-b111-47c3-b3aa-152e7e59ef34-kube-api-access-pvvzd\") pod \"60b750d2-b111-47c3-b3aa-152e7e59ef34\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") "
Oct 30 19:38:47.198036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.198002    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-tmp-dir\") pod \"60b750d2-b111-47c3-b3aa-152e7e59ef34\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") "
Oct 30 19:38:47.198212 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.198034    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-dshm\") pod \"60b750d2-b111-47c3-b3aa-152e7e59ef34\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") "
Oct 30 19:38:47.198212 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.198070    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-home\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-home\") pod \"60b750d2-b111-47c3-b3aa-152e7e59ef34\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") "
Oct 30 19:38:47.198212 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.198115    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b750d2-b111-47c3-b3aa-152e7e59ef34-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60b750d2-b111-47c3-b3aa-152e7e59ef34-cabundle-cert\") pod \"60b750d2-b111-47c3-b3aa-152e7e59ef34\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") "
Oct 30 19:38:47.198212 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.198143    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-model-cache" pod "60b750d2-b111-47c3-b3aa-152e7e59ef34" (UID: "60b750d2-b111-47c3-b3aa-152e7e59ef34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:47.198212 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.198155    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-kserve-provision-location\") pod \"60b750d2-b111-47c3-b3aa-152e7e59ef34\" (UID: \"60b750d2-b111-47c3-b3aa-152e7e59ef34\") "
Oct 30 19:38:47.198506 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.198343    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:47.198506 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.198492    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b750d2-b111-47c3-b3aa-152e7e59ef34-cabundle-cert" pod "60b750d2-b111-47c3-b3aa-152e7e59ef34" (UID: "60b750d2-b111-47c3-b3aa-152e7e59ef34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:47.198917 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.198887    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-home" pod "60b750d2-b111-47c3-b3aa-152e7e59ef34" (UID: "60b750d2-b111-47c3-b3aa-152e7e59ef34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:47.200388 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.200366    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b750d2-b111-47c3-b3aa-152e7e59ef34-tls-certs" pod "60b750d2-b111-47c3-b3aa-152e7e59ef34" (UID: "60b750d2-b111-47c3-b3aa-152e7e59ef34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:47.200776 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.200746    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-dshm" pod "60b750d2-b111-47c3-b3aa-152e7e59ef34" (UID: "60b750d2-b111-47c3-b3aa-152e7e59ef34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:47.200875 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.200828    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b750d2-b111-47c3-b3aa-152e7e59ef34-kube-api-access-pvvzd" pod "60b750d2-b111-47c3-b3aa-152e7e59ef34" (UID: "60b750d2-b111-47c3-b3aa-152e7e59ef34"). InnerVolumeSpecName "kube-api-access-pvvzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:47.214540 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.214502    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-tmp-dir" pod "60b750d2-b111-47c3-b3aa-152e7e59ef34" (UID: "60b750d2-b111-47c3-b3aa-152e7e59ef34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:47.225411 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.225389    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-kserve-provision-location" pod "60b750d2-b111-47c3-b3aa-152e7e59ef34" (UID: "60b750d2-b111-47c3-b3aa-152e7e59ef34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:47.299082 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.299028    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:47.299082 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.299050    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:47.299082 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.299058    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:47.299082 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.299066    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60b750d2-b111-47c3-b3aa-152e7e59ef34-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:47.299082 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.299076    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60b750d2-b111-47c3-b3aa-152e7e59ef34-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:47.299082 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.299085    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b750d2-b111-47c3-b3aa-152e7e59ef34-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:47.299569 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.299095    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvvzd\" (UniqueName: \"kubernetes.io/projected/60b750d2-b111-47c3-b3aa-152e7e59ef34-kube-api-access-pvvzd\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:47.964267 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.964236    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"60b750d2-b111-47c3-b3aa-152e7e59ef34","Type":"ContainerDied","Data":"d2eafe5c4d841182b50c33eae816c3cd9f947b2daf3f50cde935382989cf18ac"}
Oct 30 19:38:47.964267 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.964264    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 30 19:38:47.964452 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.964307    4853 scope.go:122] "RemoveContainer" containerID="a5ca27342e3456b68982deb0aac8d0449c4a9d4b1d1c57fbc902f3d0400f4403"
Oct 30 19:38:47.973261 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:47.973240    4853 scope.go:122] "RemoveContainer" containerID="89b17fa00881bafe38c9f66043a0c413674b95df81acd09654745655e5f4476a"
Oct 30 19:38:48.170364 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:48.170334    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 19:38:48.172463 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:48.172437    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 30 19:38:48.348799 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:48.348766    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:49.874991 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:38:49.874948    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:38:49.939633 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:49.939601    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" path="/var/lib/kubelet/pods/60b750d2-b111-47c3-b3aa-152e7e59ef34/volumes"
Oct 30 19:38:55.557991 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:55.557949    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:55.762888 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:38:55.762837    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:09.237755 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:09.237708    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:19.874095 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:39:19.874056    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:39:20.779766 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:20.779725    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:20.982487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:20.982437    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:28.106444 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:28.106389    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:31.075591 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:31.075545    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:31.282008 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:31.281978    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:33.214982 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.214952    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"]
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215232    4853 cpu_manager.go:414] "containerMap: removing container" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerName="istio-proxy"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215243    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerName="istio-proxy"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215255    4853 cpu_manager.go:414] "containerMap: removing container" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215260    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215268    4853 cpu_manager.go:414] "containerMap: removing container" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215273    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215285    4853 cpu_manager.go:414] "containerMap: removing container" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="storage-initializer"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215290    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="storage-initializer"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215297    4853 cpu_manager.go:414] "containerMap: removing container" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerName="istio-proxy"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215302    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerName="istio-proxy"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215353    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215363    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1ccf1b-7c4d-4b13-9918-0b65bcdebfd9" containerName="istio-proxy"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215369    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b750d2-b111-47c3-b3aa-152e7e59ef34" containerName="main"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215376    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7bb60a-962a-401b-8cd9-413217ba989c" containerName="istio-proxy"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215421    4853 cpu_manager.go:414] "containerMap: removing container" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy"
Oct 30 19:39:33.215429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215427    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy"
Oct 30 19:39:33.216230 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.215474    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5788ec-1675-4664-8273-8800c084b91b" containerName="istio-proxy"
Oct 30 19:39:33.220600 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.220581    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.271736 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.271707    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:39:33.271856 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.271815    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Oct 30 19:39:33.312345 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.312318    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-kserve-provision-location\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.312488 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.312361    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s979w\" (UniqueName: \"kubernetes.io/projected/f43f654b-cdca-4adc-b0bc-250b170863a6-kube-api-access-s979w\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.312488 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.312385    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-model-cache\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.312642 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.312502    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-dshm\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.312642 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.312584    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-tmp-dir\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.312642 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.312634    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-home\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.312795 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.312659    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f43f654b-cdca-4adc-b0bc-250b170863a6-cabundle-cert\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.312795 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.312685    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43f654b-cdca-4adc-b0bc-250b170863a6-tls-certs\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.317249 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.317226    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Oct 30 19:39:33.317552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.317500    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-dockercfg-9s2xh\""
Oct 30 19:39:33.317552 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.317540    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Oct 30 19:39:33.317692 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.317565    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Oct 30 19:39:33.317692 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.317576    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-72njd\""
Oct 30 19:39:33.320125 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.320105    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"]
Oct 30 19:39:33.413899 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.413870    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s979w\" (UniqueName: \"kubernetes.io/projected/f43f654b-cdca-4adc-b0bc-250b170863a6-kube-api-access-s979w\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414047 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.413904    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-model-cache\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414047 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.413925    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-dshm\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414047 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.413945    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-tmp-dir\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414047 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.413965    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-home\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414277 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.414070    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f43f654b-cdca-4adc-b0bc-250b170863a6-cabundle-cert\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414277 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.414108    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43f654b-cdca-4adc-b0bc-250b170863a6-tls-certs\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414277 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.414174    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-kserve-provision-location\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.414317    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-home\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414429 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.414416    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-model-cache\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.414457    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-tmp-dir\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.414485    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-kserve-provision-location\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.414794 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.414774    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f43f654b-cdca-4adc-b0bc-250b170863a6-cabundle-cert\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.416629 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.416605    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-dshm\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.417031 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.417007    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43f654b-cdca-4adc-b0bc-250b170863a6-tls-certs\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.469018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.468953    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s979w\" (UniqueName: \"kubernetes.io/projected/f43f654b-cdca-4adc-b0bc-250b170863a6-kube-api-access-s979w\") pod \"tls-verification-test-kserve-5998d76f5c-wwqll\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.480679 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.480657    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:33.536911 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.536886    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:33.963073 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.963040    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:33.970011 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:33.969976    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"]
Oct 30 19:39:34.010754 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:39:34.010729    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-72bb78650163f45bfc7ac6510feed4679f22cb569cf5b5bf1fcbdefd3fc41ccb WatchSource:0}: Error finding container 72bb78650163f45bfc7ac6510feed4679f22cb569cf5b5bf1fcbdefd3fc41ccb: Status 404 returned error can't find the container with id 72bb78650163f45bfc7ac6510feed4679f22cb569cf5b5bf1fcbdefd3fc41ccb
Oct 30 19:39:34.178402 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:34.178367    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" event={"ID":"f43f654b-cdca-4adc-b0bc-250b170863a6","Type":"ContainerStarted","Data":"d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b"}
Oct 30 19:39:34.178579 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:34.178422    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" event={"ID":"f43f654b-cdca-4adc-b0bc-250b170863a6","Type":"ContainerStarted","Data":"72bb78650163f45bfc7ac6510feed4679f22cb569cf5b5bf1fcbdefd3fc41ccb"}
Oct 30 19:39:34.178579 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:34.178459    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:35.184344 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:35.184308    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" event={"ID":"f43f654b-cdca-4adc-b0bc-250b170863a6","Type":"ContainerStarted","Data":"f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed"}
Oct 30 19:39:35.510018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:35.509903    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:35.712572 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:35.712512    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:36.181651 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:36.181607    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:37.198249 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:37.198216    4853 generic.go:356] "Generic (PLEG): container finished" podID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerID="f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed" exitCode=0
Oct 30 19:39:37.198751 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:37.198296    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" event={"ID":"f43f654b-cdca-4adc-b0bc-250b170863a6","Type":"ContainerDied","Data":"f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed"}
Oct 30 19:39:38.212599 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:38.212558    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" event={"ID":"f43f654b-cdca-4adc-b0bc-250b170863a6","Type":"ContainerStarted","Data":"d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840"}
Oct 30 19:39:38.376803 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:38.376748    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podStartSLOduration=3.202290416 podStartE2EDuration="5.376734827s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.174444411s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:38.37481643 +0000 UTC m=+3684.545173279" watchObservedRunningTime="2026-10-30 19:39:38.376734827 +0000 UTC m=+3684.547091674"
Oct 30 19:39:43.539554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:43.538621    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:43.543551 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:43.540818    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:43.543551 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:43.540945    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Oct 30 19:39:44.259580 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:44.259551    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:39:45.476681 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:45.476641    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:45.998366 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:45.998320    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:46.866662 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:46.866628    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:49.873022 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:39:49.872987    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:39:53.538274 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:53.538219    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Oct 30 19:39:55.301923 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.301890    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"]
Oct 30 19:39:55.304175 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.304158    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.348835 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.348802    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"]
Oct 30 19:39:55.351267 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.351248    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.404285 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404255    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.404285 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404286    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbc5n\" (UniqueName: \"kubernetes.io/projected/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-kube-api-access-gbc5n\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.404499 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404308    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.404499 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404330    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.404499 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404489    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.404688 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404515    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.404688 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404568    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx26m\" (UniqueName: \"kubernetes.io/projected/a792ece3-61ba-44d8-96e6-b0973bdd83d2-kube-api-access-gx26m\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.404688 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404663    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.404842 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404720    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.404842 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404748    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.404842 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404776    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.404842 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404827    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.405012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404866    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.405012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404889    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.405012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404940    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.405012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.404965    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.405194 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.405031    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.405194 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.405064    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.462840 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.462808    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"]
Oct 30 19:39:55.501432 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.501403    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"]
Oct 30 19:39:55.505806 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.505780    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.505806 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.505812    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.505838    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.506001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.505861    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.505889    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.505916    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.506001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.505943    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.505970    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.506310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506003    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.506310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506036    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506070    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506097    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbc5n\" (UniqueName: \"kubernetes.io/projected/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-kube-api-access-gbc5n\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506124    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.506310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506146    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.506310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506177    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506218    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506262    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.506310 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506266    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx26m\" (UniqueName: \"kubernetes.io/projected/a792ece3-61ba-44d8-96e6-b0973bdd83d2-kube-api-access-gx26m\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.506882 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506320    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.506882 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506385    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506882 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506470    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506882 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506686    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.506882 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.506781    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.507466 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.507444    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.507628 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.507605    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.507721 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.507655    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.507922 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.507882    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.508246 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.508220    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.510222 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.510191    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.510592 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.510571    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.510680 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.510571    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.515788 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.515762    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.563010 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.562928    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a792ece3-61ba-44d8-96e6-b0973bdd83d2-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.563339 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.563315    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx26m\" (UniqueName: \"kubernetes.io/projected/a792ece3-61ba-44d8-96e6-b0973bdd83d2-kube-api-access-gx26m\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-hr2xj\" (UID: \"a792ece3-61ba-44d8-96e6-b0973bdd83d2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:55.563498 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.563473    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbc5n\" (UniqueName: \"kubernetes.io/projected/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-kube-api-access-gbc5n\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.565691 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.565671    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5581ec34-b5b8-4c11-a8ae-f31fe0e894f4-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-bg5dz\" (UID: \"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.616725 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.616687    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:55.663621 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:55.663588    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:56.044158 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.044116    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"]
Oct 30 19:39:56.051549 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:39:56.051493    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5581ec34_b5b8_4c11_a8ae_f31fe0e894f4.slice/crio-b12fe9c999adc1873ec1f88628e1909171ed810c732210f410de61b06bc94a4c WatchSource:0}: Error finding container b12fe9c999adc1873ec1f88628e1909171ed810c732210f410de61b06bc94a4c: Status 404 returned error can't find the container with id b12fe9c999adc1873ec1f88628e1909171ed810c732210f410de61b06bc94a4c
Oct 30 19:39:56.053769 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.053717    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:39:56.053868 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.053815    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:39:56.053868 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.053856    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:39:56.098046 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.097965    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"]
Oct 30 19:39:56.104735 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:39:56.104708    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda792ece3_61ba_44d8_96e6_b0973bdd83d2.slice/crio-d5a9da43d77b982abf1fa077eedabd6897edc4e1ec2880320082b568ce924f18 WatchSource:0}: Error finding container d5a9da43d77b982abf1fa077eedabd6897edc4e1ec2880320082b568ce924f18: Status 404 returned error can't find the container with id d5a9da43d77b982abf1fa077eedabd6897edc4e1ec2880320082b568ce924f18
Oct 30 19:39:56.107746 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.107703    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:39:56.107857 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.107798    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:39:56.107857 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.107847    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:39:56.308290 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.307578    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj" event={"ID":"a792ece3-61ba-44d8-96e6-b0973bdd83d2","Type":"ContainerStarted","Data":"04e82d68818ca1a8941613dcccb02c39ea9bb8c76288bdb30f1779c78afd28bb"}
Oct 30 19:39:56.308290 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.307650    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj" event={"ID":"a792ece3-61ba-44d8-96e6-b0973bdd83d2","Type":"ContainerStarted","Data":"d5a9da43d77b982abf1fa077eedabd6897edc4e1ec2880320082b568ce924f18"}
Oct 30 19:39:56.309642 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.309615    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz" event={"ID":"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4","Type":"ContainerStarted","Data":"54c94a655f6817ab42b80ead741afd1f9e73789b3b3cd90314c957a50cb5bf4e"}
Oct 30 19:39:56.309783 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.309663    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz" event={"ID":"5581ec34-b5b8-4c11-a8ae-f31fe0e894f4","Type":"ContainerStarted","Data":"b12fe9c999adc1873ec1f88628e1909171ed810c732210f410de61b06bc94a4c"}
Oct 30 19:39:56.420547 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.420476    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj" podStartSLOduration=1.42046161 podStartE2EDuration="1.42046161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:56.419896753 +0000 UTC m=+3702.590253615" watchObservedRunningTime="2026-10-30 19:39:56.42046161 +0000 UTC m=+3702.590818498"
Oct 30 19:39:56.536110 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.532838    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz" podStartSLOduration=1.532817345 podStartE2EDuration="1.532817345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:56.532081017 +0000 UTC m=+3702.702437886" watchObservedRunningTime="2026-10-30 19:39:56.532817345 +0000 UTC m=+3702.703174197"
Oct 30 19:39:56.619553 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.618626    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:39:56.669656 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:56.669618    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:39:57.318900 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:57.318843    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:57.538559 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:57.538390    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:57.618567 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:57.618383    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:39:57.618567 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:57.618435    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz" podUID="5581ec34-b5b8-4c11-a8ae-f31fe0e894f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:39:57.673567 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:57.672670    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:39:57.673567 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:57.672754    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj" podUID="a792ece3-61ba-44d8-96e6-b0973bdd83d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:39:58.617613 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:58.617577    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:39:58.618015 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:58.617631    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz" podUID="5581ec34-b5b8-4c11-a8ae-f31fe0e894f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:39:58.664938 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:58.664900    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:39:58.665195 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:58.665170    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj" podUID="a792ece3-61ba-44d8-96e6-b0973bdd83d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:39:59.617597 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:59.617554    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:39:59.617793 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:59.617613    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz" podUID="5581ec34-b5b8-4c11-a8ae-f31fe0e894f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:39:59.665554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:59.664882    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:39:59.665554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:59.664943    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj" podUID="a792ece3-61ba-44d8-96e6-b0973bdd83d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:39:59.956749 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:39:59.956488    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:00.618072 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:00.618035    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:40:00.618487 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:00.618090    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz" podUID="5581ec34-b5b8-4c11-a8ae-f31fe0e894f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:40:00.666030 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:00.665906    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:40:00.666030 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:00.665971    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj" podUID="a792ece3-61ba-44d8-96e6-b0973bdd83d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:40:01.309496 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:01.309456    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:01.617714 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:01.617686    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:40:01.617960 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:01.617939    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz" podUID="5581ec34-b5b8-4c11-a8ae-f31fe0e894f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 19:40:01.665921 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:01.665883    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:40:01.666558 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:01.666502    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj" podUID="a792ece3-61ba-44d8-96e6-b0973bdd83d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 19:40:02.621176 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:02.621147    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:40:02.621467 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:02.621444    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:40:02.622089 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:02.622071    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-bg5dz"
Oct 30 19:40:02.668831 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:02.668807    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:40:02.669229 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:02.669085    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:40:02.669803 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:02.669784    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-hr2xj"
Oct 30 19:40:03.538197 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:03.538150    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Oct 30 19:40:06.336842 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:06.336801    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:09.338089 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:09.338041    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:09.541093 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:09.540976    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:10.407563 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.407507    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"]
Oct 30 19:40:10.411037 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.411010    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.490691 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.490661    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.490847 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.490705    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.490847 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.490724    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tz8\" (UniqueName: \"kubernetes.io/projected/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-kube-api-access-b6tz8\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.490847 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.490787    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.491026 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.490943    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.491026 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.491009    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.491152 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.491036    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.491206 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.491180    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.491251 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.491206    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.500764 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.500742    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"]
Oct 30 19:40:10.592490 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.592462    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.592659 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.592493    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.592659 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.592553    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.592659 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.592579    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.592659 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.592614    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.592659 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.592646    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.592934 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.592669    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tz8\" (UniqueName: \"kubernetes.io/projected/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-kube-api-access-b6tz8\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.592934 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.592698    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.592934 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.592729    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.593336 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.592987    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.593336 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.593095    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.593336 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.593175    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.593336 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.593241    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.593336 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.593324    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.595434 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.595414    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.595610 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.595591    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.647548 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.647499    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tz8\" (UniqueName: \"kubernetes.io/projected/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-kube-api-access-b6tz8\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.647665 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.647550    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-qz2wk\" (UID: \"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:10.723486 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:10.723420    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:11.158500 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:11.158464    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"]
Oct 30 19:40:11.166583 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:40:11.166552    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e960a4_b8fb_4bf9_a6e6_e8f02b3aa42d.slice/crio-ade6261a9c490ce1830b57d9306453d5e677e55519e5120bd0ac7acf3ca30aa2 WatchSource:0}: Error finding container ade6261a9c490ce1830b57d9306453d5e677e55519e5120bd0ac7acf3ca30aa2: Status 404 returned error can't find the container with id ade6261a9c490ce1830b57d9306453d5e677e55519e5120bd0ac7acf3ca30aa2
Oct 30 19:40:11.169287 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:11.169253    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:40:11.169343 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:11.169329    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:40:11.169386 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:11.169358    4853 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:40:11.399211 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:11.399164    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk" event={"ID":"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d","Type":"ContainerStarted","Data":"a164b3820e4d4d5232d9fd1f97d344073ebfde11a87b5eee85e7eb4bec60cafd"}
Oct 30 19:40:11.399375 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:11.399228    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk" event={"ID":"a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d","Type":"ContainerStarted","Data":"ade6261a9c490ce1830b57d9306453d5e677e55519e5120bd0ac7acf3ca30aa2"}
Oct 30 19:40:11.519328 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:11.519233    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk" podStartSLOduration=1.519219112 podStartE2EDuration="1.519219112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:11.517128903 +0000 UTC m=+3717.687485751" watchObservedRunningTime="2026-10-30 19:40:11.519219112 +0000 UTC m=+3717.689575959"
Oct 30 19:40:11.724227 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:11.724188    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:12.725508 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:12.725474    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:40:12.726083 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:12.725549    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk" podUID="a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:40:13.319493 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:13.319459    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:13.541551 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:13.539930    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Oct 30 19:40:13.609116 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:13.609077    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:13.724074 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:13.724040    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:40:13.724250 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:13.724091    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk" podUID="a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:40:14.644435 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:14.644396    4853 scope.go:122] "RemoveContainer" containerID="444109a996d06b4f400d56a12fbb54af31ba005abf35518cf824da9313792856"
Oct 30 19:40:14.723773 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:14.723744    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:40:14.723876 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:14.723794    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk" podUID="a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:40:15.724146 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:15.724116    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:40:15.724572 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:15.724168    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk" podUID="a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:40:16.723931 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:16.723896    4853 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:40:16.724114 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:16.723950    4853 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk" podUID="a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 19:40:17.727404 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:17.727369    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:17.727949 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:17.727784    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:17.728438 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:17.728421    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-qz2wk"
Oct 30 19:40:19.877495 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:40:19.877455    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:40:23.538004 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:23.537951    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Oct 30 19:40:33.537395 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:33.537340    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Oct 30 19:40:43.537646 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:43.537592    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Oct 30 19:40:49.872673 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:40:49.872635    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:40:53.537612 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:40:53.537573    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Oct 30 19:41:03.537693 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:41:03.537635    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Oct 30 19:41:09.736739 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:41:09.736697    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:13.537409 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:41:13.537355    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Oct 30 19:41:19.872256 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:41:19.872220    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:41:23.538184 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:41:23.538133    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8001/health\": dial tcp 10.134.0.76:8001: connect: connection refused"
Oct 30 19:41:30.401499 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:41:30.401457    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:33.547704 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:41:33.547676    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:41:33.559741 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:41:33.559715    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:41:49.876908 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:41:49.876869    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:42:10.082255 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:10.082220    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:12.819253 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:12.819219    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:19.216598 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.216559    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"]
Oct 30 19:42:19.220088 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.220065    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.270663 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.270636    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Oct 30 19:42:19.270774 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.270707    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:42:19.301036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.300993    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kserve-provision-location\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.301231 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.301192    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tls-certs\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.301306 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.301250    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2x5\" (UniqueName: \"kubernetes.io/projected/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kube-api-access-mv2x5\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.301381 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.301353    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-home\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.301522 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.301411    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-cabundle-cert\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.301643 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.301523    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-dshm\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.301643 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.301617    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-model-cache\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.301757 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.301658    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tmp-dir\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.316471 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.316444    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Oct 30 19:42:19.316638 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.316595    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Oct 30 19:42:19.316720 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.316667    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Oct 30 19:42:19.316720 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.316691    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-m94j9\""
Oct 30 19:42:19.318646 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.318631    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"]
Oct 30 19:42:19.402876 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.402849    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2x5\" (UniqueName: \"kubernetes.io/projected/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kube-api-access-mv2x5\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403003 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.402878    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-home\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403003 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.402901    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-cabundle-cert\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403003 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.402920    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-dshm\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403003 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.402941    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-model-cache\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403003 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.402964    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tmp-dir\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403315 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.402988    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kserve-provision-location\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403426 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.403406    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-home\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403498 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.403482    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-model-cache\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403563 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.403502    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kserve-provision-location\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403616 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.403590    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tmp-dir\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403711 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.403635    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tls-certs\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.403770 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.403753    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-cabundle-cert\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.405581 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.405558    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-dshm\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.406173 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.406153    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tls-certs\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.458588 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.458560    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2x5\" (UniqueName: \"kubernetes.io/projected/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kube-api-access-mv2x5\") pod \"prestop-hook-test-kserve-698749d5d6-pn6qn\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.535603 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.535548    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:19.872577 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:42:19.872514    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:42:19.967558 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:19.967513    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"]
Oct 30 19:42:19.973266 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:42:19.973239    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fa6e3e_395d_4f16_9ed5_d9efe31dbda3.slice/crio-d29f0f89325d3c23c31bf4bd2e18cb2e569c197a48fdbf8028687d27294866a7 WatchSource:0}: Error finding container d29f0f89325d3c23c31bf4bd2e18cb2e569c197a48fdbf8028687d27294866a7: Status 404 returned error can't find the container with id d29f0f89325d3c23c31bf4bd2e18cb2e569c197a48fdbf8028687d27294866a7
Oct 30 19:42:20.018166 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:20.018141    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn" event={"ID":"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3","Type":"ContainerStarted","Data":"d29f0f89325d3c23c31bf4bd2e18cb2e569c197a48fdbf8028687d27294866a7"}
Oct 30 19:42:21.024651 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:21.024616    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn" event={"ID":"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3","Type":"ContainerStarted","Data":"acacf6c1f1c92fc830dd1b601e9fc5fb2f3b293c02a6bc5b682a81c93c4feafb"}
Oct 30 19:42:23.035399 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:23.035366    4853 generic.go:356] "Generic (PLEG): container finished" podID="e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" containerID="acacf6c1f1c92fc830dd1b601e9fc5fb2f3b293c02a6bc5b682a81c93c4feafb" exitCode=0
Oct 30 19:42:23.035844 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:23.035414    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn" event={"ID":"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3","Type":"ContainerDied","Data":"acacf6c1f1c92fc830dd1b601e9fc5fb2f3b293c02a6bc5b682a81c93c4feafb"}
Oct 30 19:42:24.046540 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:24.046487    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn" event={"ID":"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3","Type":"ContainerStarted","Data":"91f5ef70387c9a8cde94eb3320ba23b0956787f47fd73ac078c91b1838c11520"}
Oct 30 19:42:24.255427 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:24.255364    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn" podStartSLOduration=3.100716234 podStartE2EDuration="5.255345561s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.154629327s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:24.206574453 +0000 UTC m=+3850.376931300" watchObservedRunningTime="2026-10-30 19:42:24.255345561 +0000 UTC m=+3850.425702413"
Oct 30 19:42:24.535658 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:24.535627    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:29.536005 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:29.535967    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:42:29.537430 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:29.537399    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn" podUID="e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Oct 30 19:42:39.536039 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:39.535991    4853 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn" podUID="e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Oct 30 19:42:40.811759 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:40.811726    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"]
Oct 30 19:42:40.866400 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:40.866359    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:41.079337 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:41.079290    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:49.873120 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:42:49.873073    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:42:55.843372 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:42:55.843326    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn" podUID="e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" containerName="main" containerID="cri-o://91f5ef70387c9a8cde94eb3320ba23b0956787f47fd73ac078c91b1838c11520" gracePeriod=45
Oct 30 19:43:10.431004 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:10.430966    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:17.686342 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:17.686302    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"]
Oct 30 19:43:17.713592 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:17.713557    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:17.965686 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:17.965604    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:19.898515 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:43:19.898472    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:43:30.018047 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:30.018005    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:30.223762 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:30.223728    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:32.710473 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:32.710428    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" containerID="cri-o://d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840" gracePeriod=45
Oct 30 19:43:33.528459 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.528428    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="llm-d-routing-sidecar" containerID="cri-o://d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b" gracePeriod=45
Oct 30 19:43:33.537216 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.537185    4853 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 19:43:33.548766 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.548741    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 19:43:33.777711 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.777687    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:43:33.956304 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.956276    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-home\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-home\") pod \"f43f654b-cdca-4adc-b0bc-250b170863a6\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") "
Oct 30 19:43:33.956458 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.956310    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43f654b-cdca-4adc-b0bc-250b170863a6-kube-api-access-s979w\" (UniqueName: \"kubernetes.io/projected/f43f654b-cdca-4adc-b0bc-250b170863a6-kube-api-access-s979w\") pod \"f43f654b-cdca-4adc-b0bc-250b170863a6\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") "
Oct 30 19:43:33.956458 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.956351    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f43f654b-cdca-4adc-b0bc-250b170863a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/f43f654b-cdca-4adc-b0bc-250b170863a6-tls-certs\") pod \"f43f654b-cdca-4adc-b0bc-250b170863a6\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") "
Oct 30 19:43:33.956458 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.956368    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-kserve-provision-location\") pod \"f43f654b-cdca-4adc-b0bc-250b170863a6\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") "
Oct 30 19:43:33.956458 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.956392    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-model-cache\") pod \"f43f654b-cdca-4adc-b0bc-250b170863a6\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") "
Oct 30 19:43:33.956458 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.956414    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-dshm\") pod \"f43f654b-cdca-4adc-b0bc-250b170863a6\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") "
Oct 30 19:43:33.956851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.956465    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43f654b-cdca-4adc-b0bc-250b170863a6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f43f654b-cdca-4adc-b0bc-250b170863a6-cabundle-cert\") pod \"f43f654b-cdca-4adc-b0bc-250b170863a6\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") "
Oct 30 19:43:33.956851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.956509    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-tmp-dir\") pod \"f43f654b-cdca-4adc-b0bc-250b170863a6\" (UID: \"f43f654b-cdca-4adc-b0bc-250b170863a6\") "
Oct 30 19:43:33.957129 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.957094    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-home" pod "f43f654b-cdca-4adc-b0bc-250b170863a6" (UID: "f43f654b-cdca-4adc-b0bc-250b170863a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:33.957229 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.957182    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43f654b-cdca-4adc-b0bc-250b170863a6-cabundle-cert" pod "f43f654b-cdca-4adc-b0bc-250b170863a6" (UID: "f43f654b-cdca-4adc-b0bc-250b170863a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:33.957304 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.957245    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-model-cache" pod "f43f654b-cdca-4adc-b0bc-250b170863a6" (UID: "f43f654b-cdca-4adc-b0bc-250b170863a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:33.959209 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.959184    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-dshm" pod "f43f654b-cdca-4adc-b0bc-250b170863a6" (UID: "f43f654b-cdca-4adc-b0bc-250b170863a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:33.959313 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.959237    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43f654b-cdca-4adc-b0bc-250b170863a6-tls-certs" pod "f43f654b-cdca-4adc-b0bc-250b170863a6" (UID: "f43f654b-cdca-4adc-b0bc-250b170863a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:33.959378 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.959325    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43f654b-cdca-4adc-b0bc-250b170863a6-kube-api-access-s979w" pod "f43f654b-cdca-4adc-b0bc-250b170863a6" (UID: "f43f654b-cdca-4adc-b0bc-250b170863a6"). InnerVolumeSpecName "kube-api-access-s979w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:33.970021 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.969997    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-tmp-dir" pod "f43f654b-cdca-4adc-b0bc-250b170863a6" (UID: "f43f654b-cdca-4adc-b0bc-250b170863a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:33.981253 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:33.981218    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-kserve-provision-location" pod "f43f654b-cdca-4adc-b0bc-250b170863a6" (UID: "f43f654b-cdca-4adc-b0bc-250b170863a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:34.057620 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.057566    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:34.057620 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.057588    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s979w\" (UniqueName: \"kubernetes.io/projected/f43f654b-cdca-4adc-b0bc-250b170863a6-kube-api-access-s979w\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:34.057620 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.057597    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43f654b-cdca-4adc-b0bc-250b170863a6-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:34.057620 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.057606    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:34.057620 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.057614    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:34.057851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.057623    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:34.057851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.057632    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f43f654b-cdca-4adc-b0bc-250b170863a6-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:34.057851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.057639    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43f654b-cdca-4adc-b0bc-250b170863a6-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:34.400246 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.400219    4853 generic.go:356] "Generic (PLEG): container finished" podID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerID="d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840" exitCode=0
Oct 30 19:43:34.400246 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.400242    4853 generic.go:356] "Generic (PLEG): container finished" podID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerID="d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b" exitCode=0
Oct 30 19:43:34.400433 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.400283    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" event={"ID":"f43f654b-cdca-4adc-b0bc-250b170863a6","Type":"ContainerDied","Data":"d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840"}
Oct 30 19:43:34.400433 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.400321    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" event={"ID":"f43f654b-cdca-4adc-b0bc-250b170863a6","Type":"ContainerDied","Data":"d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b"}
Oct 30 19:43:34.400433 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.400336    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll" event={"ID":"f43f654b-cdca-4adc-b0bc-250b170863a6","Type":"ContainerDied","Data":"72bb78650163f45bfc7ac6510feed4679f22cb569cf5b5bf1fcbdefd3fc41ccb"}
Oct 30 19:43:34.400433 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.400357    4853 scope.go:122] "RemoveContainer" containerID="d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840"
Oct 30 19:43:34.400433 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.400282    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"
Oct 30 19:43:34.410470 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.410452    4853 scope.go:122] "RemoveContainer" containerID="f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed"
Oct 30 19:43:34.418388 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.418367    4853 scope.go:122] "RemoveContainer" containerID="d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b"
Oct 30 19:43:34.427330 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.427305    4853 scope.go:122] "RemoveContainer" containerID="d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840"
Oct 30 19:43:34.427624 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:43:34.427608    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840\": container with ID starting with d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840 not found: ID does not exist" containerID="d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840"
Oct 30 19:43:34.427691 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.427632    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840"} err="failed to get container status \"d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840\": rpc error: code = NotFound desc = could not find container \"d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840\": container with ID starting with d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840 not found: ID does not exist"
Oct 30 19:43:34.427691 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.427649    4853 scope.go:122] "RemoveContainer" containerID="f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed"
Oct 30 19:43:34.427884 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:43:34.427858    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed\": container with ID starting with f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed not found: ID does not exist" containerID="f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed"
Oct 30 19:43:34.427924 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.427896    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed"} err="failed to get container status \"f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed\": rpc error: code = NotFound desc = could not find container \"f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed\": container with ID starting with f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed not found: ID does not exist"
Oct 30 19:43:34.427924 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.427917    4853 scope.go:122] "RemoveContainer" containerID="d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b"
Oct 30 19:43:34.428186 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:43:34.428168    4853 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b\": container with ID starting with d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b not found: ID does not exist" containerID="d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b"
Oct 30 19:43:34.428246 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.428191    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b"} err="failed to get container status \"d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b\": rpc error: code = NotFound desc = could not find container \"d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b\": container with ID starting with d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b not found: ID does not exist"
Oct 30 19:43:34.428246 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.428208    4853 scope.go:122] "RemoveContainer" containerID="d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840"
Oct 30 19:43:34.428442 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.428423    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840"} err="failed to get container status \"d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840\": rpc error: code = NotFound desc = could not find container \"d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840\": container with ID starting with d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840 not found: ID does not exist"
Oct 30 19:43:34.428543 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.428442    4853 scope.go:122] "RemoveContainer" containerID="f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed"
Oct 30 19:43:34.428696 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.428674    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed"} err="failed to get container status \"f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed\": rpc error: code = NotFound desc = could not find container \"f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed\": container with ID starting with f92e9861a5919344df4c42d3c1f6e5744cd396c65758940c8849749ccbaebfed not found: ID does not exist"
Oct 30 19:43:34.428774 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.428699    4853 scope.go:122] "RemoveContainer" containerID="d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b"
Oct 30 19:43:34.428925 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.428904    4853 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b"} err="failed to get container status \"d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b\": rpc error: code = NotFound desc = could not find container \"d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b\": container with ID starting with d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b not found: ID does not exist"
Oct 30 19:43:34.562420 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.562391    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"]
Oct 30 19:43:34.568915 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:34.568890    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5998d76f5c-wwqll"]
Oct 30 19:43:35.938999 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:35.938963    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" path="/var/lib/kubelet/pods/f43f654b-cdca-4adc-b0bc-250b170863a6/volumes"
Oct 30 19:43:37.239753 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:43:37.239723    4853 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/fe96d8d4c38c5959b824b86fabaf3a9bec7a0ea19b62100aacc94e491e46a296/diff" to get inode usage: stat /var/lib/containers/storage/overlay/fe96d8d4c38c5959b824b86fabaf3a9bec7a0ea19b62100aacc94e491e46a296/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/e2e-test-llm-tls-resources-e4cad0c3_tls-verification-test-kserve-5998d76f5c-wwqll_f43f654b-cdca-4adc-b0bc-250b170863a6/main/0.log" to get inode usage: stat /var/log/pods/e2e-test-llm-tls-resources-e4cad0c3_tls-verification-test-kserve-5998d76f5c-wwqll_f43f654b-cdca-4adc-b0bc-250b170863a6/main/0.log: no such file or directory
Oct 30 19:43:40.864780 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:43:40.864734    4853 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-conmon-d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-72bb78650163f45bfc7ac6510feed4679f22cb569cf5b5bf1fcbdefd3fc41ccb\": RecentStats: unable to find data in memory cache]"
Oct 30 19:43:40.865209 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:43:40.864992    4853 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-conmon-d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-72bb78650163f45bfc7ac6510feed4679f22cb569cf5b5bf1fcbdefd3fc41ccb\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-conmon-d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:43:40.865209 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:43:40.865017    4853 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-conmon-d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-conmon-d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-72bb78650163f45bfc7ac6510feed4679f22cb569cf5b5bf1fcbdefd3fc41ccb\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:43:40.868948 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:43:40.868908    4853 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-72bb78650163f45bfc7ac6510feed4679f22cb569cf5b5bf1fcbdefd3fc41ccb\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-conmon-d5c57c89a009ca3d205e094caea22ea646018a8bcfd976ece6b8ae7bb5023840.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43f654b_cdca_4adc_b0bc_250b170863a6.slice/crio-conmon-d3f6f5fa2467be584e2c788dd62036732cc4a63b1b85c7ba4b9872fbbaa9144b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:43:41.436043 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.436018    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-698749d5d6-pn6qn_e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3/main/0.log"
Oct 30 19:43:41.436470 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.436439    4853 generic.go:356] "Generic (PLEG): container finished" podID="e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" containerID="91f5ef70387c9a8cde94eb3320ba23b0956787f47fd73ac078c91b1838c11520" exitCode=137
Oct 30 19:43:41.436618 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.436488    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn" event={"ID":"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3","Type":"ContainerDied","Data":"91f5ef70387c9a8cde94eb3320ba23b0956787f47fd73ac078c91b1838c11520"}
Oct 30 19:43:41.498684 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.498665    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-698749d5d6-pn6qn_e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3/main/0.log"
Oct 30 19:43:41.501282 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.501262    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:43:41.613267 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.613214    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-cabundle-cert\") pod \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") "
Oct 30 19:43:41.613267 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.613247    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-home\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-home\") pod \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") "
Oct 30 19:43:41.613441 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.613273    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kube-api-access-mv2x5\" (UniqueName: \"kubernetes.io/projected/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kube-api-access-mv2x5\") pod \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") "
Oct 30 19:43:41.613441 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.613294    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-model-cache\") pod \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") "
Oct 30 19:43:41.613441 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.613363    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kserve-provision-location\") pod \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") "
Oct 30 19:43:41.613441 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.613422    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tmp-dir\") pod \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") "
Oct 30 19:43:41.613691 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.613454    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-dshm\") pod \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") "
Oct 30 19:43:41.613691 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.613502    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tls-certs\") pod \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\" (UID: \"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3\") "
Oct 30 19:43:41.613691 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.613662    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-model-cache" pod "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" (UID: "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:41.613845 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.613761    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-cabundle-cert" pod "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" (UID: "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:41.613845 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.613814    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:41.614064 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.614038    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-home" pod "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" (UID: "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:41.614198 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.614129    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tmp-dir" pod "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" (UID: "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:41.615806 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.615782    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kube-api-access-mv2x5" pod "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" (UID: "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3"). InnerVolumeSpecName "kube-api-access-mv2x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:41.615918 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.615897    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-dshm" pod "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" (UID: "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:41.616097 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.616080    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tls-certs" pod "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" (UID: "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:41.632001 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.631979    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kserve-provision-location" pod "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" (UID: "e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:41.714568 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.714544    4853 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-cabundle-cert\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:41.714671 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.714569    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:41.714671 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.714584    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv2x5\" (UniqueName: \"kubernetes.io/projected/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kube-api-access-mv2x5\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:41.714671 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.714599    4853 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-kserve-provision-location\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:41.714671 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.714612    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:41.714671 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.714624    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:41.714671 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:41.714639    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:42.443061 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:42.443035    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-698749d5d6-pn6qn_e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3/main/0.log"
Oct 30 19:43:42.443558 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:42.443505    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn" event={"ID":"e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3","Type":"ContainerDied","Data":"d29f0f89325d3c23c31bf4bd2e18cb2e569c197a48fdbf8028687d27294866a7"}
Oct 30 19:43:42.443705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:42.443574    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"
Oct 30 19:43:42.443705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:42.443597    4853 scope.go:122] "RemoveContainer" containerID="91f5ef70387c9a8cde94eb3320ba23b0956787f47fd73ac078c91b1838c11520"
Oct 30 19:43:42.453097 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:42.453078    4853 scope.go:122] "RemoveContainer" containerID="acacf6c1f1c92fc830dd1b601e9fc5fb2f3b293c02a6bc5b682a81c93c4feafb"
Oct 30 19:43:42.607626 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:42.607596    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"]
Oct 30 19:43:42.618044 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:42.618012    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-pn6qn"]
Oct 30 19:43:43.570235 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:43.570192    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:43.939593 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:43.939561    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" path="/var/lib/kubelet/pods/e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3/volumes"
Oct 30 19:43:49.876213 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:43:49.876168    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:43:50.694201 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:50.694161    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:56.596303 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:56.596258    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:57.121542 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:57.121505    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:57.388303 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:43:57.388227    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:07.254355 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:07.254322    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:07.479441 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:07.479380    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:10.818132 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:10.818096    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:12.846518 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846487    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"]
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846816    4853 cpu_manager.go:414] "containerMap: removing container" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846828    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846837    4853 cpu_manager.go:414] "containerMap: removing container" podUID="e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" containerName="main"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846842    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" containerName="main"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846853    4853 cpu_manager.go:414] "containerMap: removing container" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="storage-initializer"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846858    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="storage-initializer"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846869    4853 cpu_manager.go:414] "containerMap: removing container" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="llm-d-routing-sidecar"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846873    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="llm-d-routing-sidecar"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846880    4853 cpu_manager.go:414] "containerMap: removing container" podUID="e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" containerName="storage-initializer"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846884    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" containerName="storage-initializer"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846933    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="main"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846941    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0fa6e3e-395d-4f16-9ed5-d9efe31dbda3" containerName="main"
Oct 30 19:44:12.847018 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.846947    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43f654b-cdca-4adc-b0bc-250b170863a6" containerName="llm-d-routing-sidecar"
Oct 30 19:44:12.849784 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.849769    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:12.862053 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.862031    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"]
Oct 30 19:44:12.901651 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.901629    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Oct 30 19:44:12.901651 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.901629    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Oct 30 19:44:12.902368 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.902351    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-x7hxk\""
Oct 30 19:44:12.902470 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.902456    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Oct 30 19:44:12.910521 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.910492    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:12.945836 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.945812    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:12.945944 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.945846    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:12.945944 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.945867    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzr7k\" (UniqueName: \"kubernetes.io/projected/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-kube-api-access-jzr7k\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:12.945944 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.945890    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:12.946083 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.945976    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:12.946083 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:12.946011    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.046966 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.046934    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.046966 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.046972    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzr7k\" (UniqueName: \"kubernetes.io/projected/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-kube-api-access-jzr7k\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.047202 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.047010    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.047202 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.047042    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.047202 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.047086    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.047202 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.047141    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.047507 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.047483    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.047507 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.047498    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.047641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.047524    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.049387 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.049366    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.049681 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.049664    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.102857 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.102795    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzr7k\" (UniqueName: \"kubernetes.io/projected/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-kube-api-access-jzr7k\") pod \"rolling-upgrade-test-kserve-5d77dc844-x2jx6\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.162654 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.162634    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:13.382807 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.382720    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:13.536091 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.536041    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"]
Oct 30 19:44:13.543006 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:44:13.542976    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5f87fc_76fc_4935_a6d5_8a36fef61d0a.slice/crio-39e57396eb41271fa516b1586ff8b53148f75dc49418ae0edb5e8c0b0e5f160b WatchSource:0}: Error finding container 39e57396eb41271fa516b1586ff8b53148f75dc49418ae0edb5e8c0b0e5f160b: Status 404 returned error can't find the container with id 39e57396eb41271fa516b1586ff8b53148f75dc49418ae0edb5e8c0b0e5f160b
Oct 30 19:44:13.597352 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.597329    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6" event={"ID":"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a","Type":"ContainerStarted","Data":"1abc184b31f112b4b83abe6d017afe102e395672735d280016f973f0727b5b82"}
Oct 30 19:44:13.597436 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:13.597377    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6" event={"ID":"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a","Type":"ContainerStarted","Data":"39e57396eb41271fa516b1586ff8b53148f75dc49418ae0edb5e8c0b0e5f160b"}
Oct 30 19:44:14.712797 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:14.712754    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6" podStartSLOduration=2.712741443 podStartE2EDuration="2.712741443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:14.712051423 +0000 UTC m=+3960.882408275" watchObservedRunningTime="2026-10-30 19:44:14.712741443 +0000 UTC m=+3960.883098291"
Oct 30 19:44:15.163164 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:15.163126    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:15.534711 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:15.534638    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:20.003556 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:44:20.002772    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:44:21.564556 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:21.563628    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:22.114554 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:22.113991    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:22.341556 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:22.340659    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:23.162758 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:23.162729    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:23.175278 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:23.175254    4853 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:23.739306 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:23.739270    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:49.873259 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:44:49.873224    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:44:54.425705 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:54.425673    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"]
Oct 30 19:44:54.435516 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:44:54.435479    4853 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="1abc184b31f112b4b83abe6d017afe102e395672735d280016f973f0727b5b82" cmd=["/bin/sleep","15"]
Oct 30 19:44:54.435690 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:54.435544    4853 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:44:54.435690 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:44:54.435559    4853 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6" podUID="5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" containerName="main" containerID="cri-o://1abc184b31f112b4b83abe6d017afe102e395672735d280016f973f0727b5b82"
Oct 30 19:44:54.435690 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:54.435604    4853 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6" podUID="5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" containerName="main" containerID="cri-o://1abc184b31f112b4b83abe6d017afe102e395672735d280016f973f0727b5b82" gracePeriod=60
Oct 30 19:44:54.512753 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:54.512724    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:54.724198 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:54.724105    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:54.727926 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:54.727886    4853 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6" podUID="5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": dial tcp 10.134.0.81:8000: connect: connection refused"
Oct 30 19:44:54.871558 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:54.871510    4853 generic.go:356] "Generic (PLEG): container finished" podID="5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" containerID="1abc184b31f112b4b83abe6d017afe102e395672735d280016f973f0727b5b82" exitCode=0
Oct 30 19:44:54.871695 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:54.871583    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6" event={"ID":"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a","Type":"ContainerDied","Data":"1abc184b31f112b4b83abe6d017afe102e395672735d280016f973f0727b5b82"}
Oct 30 19:44:55.198444 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.198420    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:55.256428 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.256404    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-dshm\") pod \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") "
Oct 30 19:44:55.256571 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.256439    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-model-cache\") pod \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") "
Oct 30 19:44:55.256571 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.256458    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-home\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-home\") pod \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") "
Oct 30 19:44:55.256571 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.256485    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-kube-api-access-jzr7k\" (UniqueName: \"kubernetes.io/projected/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-kube-api-access-jzr7k\") pod \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") "
Oct 30 19:44:55.256571 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.256504    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tmp-dir\") pod \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") "
Oct 30 19:44:55.256571 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.256563    4853 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tls-certs\") pod \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\" (UID: \"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a\") "
Oct 30 19:44:55.256851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.256738    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-model-cache" pod "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" (UID: "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:55.256851 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.256758    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-home" pod "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" (UID: "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:55.256958 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.256867    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tmp-dir" pod "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" (UID: "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:55.257011 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.256962    4853 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-model-cache\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:55.257011 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.256982    4853 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-home\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:55.258841 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.258817    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-kube-api-access-jzr7k" pod "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" (UID: "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a"). InnerVolumeSpecName "kube-api-access-jzr7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:55.258940 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.258856    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-dshm" pod "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" (UID: "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:55.258940 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.258869    4853 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tls-certs" pod "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" (UID: "5c5f87fc-76fc-4935-a6d5-8a36fef61d0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:55.357872 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.357849    4853 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-dshm\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:55.357872 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.357870    4853 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzr7k\" (UniqueName: \"kubernetes.io/projected/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-kube-api-access-jzr7k\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:55.358019 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.357880    4853 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tmp-dir\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:55.358019 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.357888    4853 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a-tls-certs\") on node \"ip-10-0-129-31.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:55.878047 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.878020    4853 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"
Oct 30 19:44:55.878518 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.878064    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6" event={"ID":"5c5f87fc-76fc-4935-a6d5-8a36fef61d0a","Type":"ContainerDied","Data":"39e57396eb41271fa516b1586ff8b53148f75dc49418ae0edb5e8c0b0e5f160b"}
Oct 30 19:44:55.878518 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:55.878120    4853 scope.go:122] "RemoveContainer" containerID="1abc184b31f112b4b83abe6d017afe102e395672735d280016f973f0727b5b82"
Oct 30 19:44:56.038086 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:56.038056    4853 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"]
Oct 30 19:44:56.084581 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:56.084554    4853 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-x2jx6"]
Oct 30 19:44:57.938662 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:44:57.938628    4853 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" path="/var/lib/kubelet/pods/5c5f87fc-76fc-4935-a6d5-8a36fef61d0a/volumes"
Oct 30 19:45:11.115291 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:45:11.115243    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:19.876841 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:45:19.876791    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:45:43.153810 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:45:43.153773    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:45.357668 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:45:45.357633    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:49.372127 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:45:49.372088    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:49.591238 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:45:49.591207    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:49.874217 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:45:49.874183    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:45:50.600703 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:45:50.600664    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:56.957295 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:45:56.957254    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:57.154829 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:45:57.154793    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:11.451341 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:46:11.451303    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:19.873291 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:46:19.873255    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:46:49.875648 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:46:49.875594    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:47:02.592030 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:02.591997    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kpb9g_7cbc10ed-835f-4b67-a55a-4bae7bf054ab/discovery/0.log"
Oct 30 19:47:03.676641 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:03.676603    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-4wgl6_82dc1cff-278b-4772-b4ef-661fc700bce7/istio-proxy/0.log"
Oct 30 19:47:03.888849 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:03.888819    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-bg5dz_5581ec34-b5b8-4c11-a8ae-f31fe0e894f4/istio-proxy/0.log"
Oct 30 19:47:04.718792 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:04.718757    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-hr2xj_a792ece3-61ba-44d8-96e6-b0973bdd83d2/istio-proxy/0.log"
Oct 30 19:47:04.931264 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:04.931230    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-qz2wk_a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d/istio-proxy/0.log"
Oct 30 19:47:08.641463 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:08.641422    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kpb9g_7cbc10ed-835f-4b67-a55a-4bae7bf054ab/discovery/0.log"
Oct 30 19:47:09.315027 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:09.314996    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-4wgl6_82dc1cff-278b-4772-b4ef-661fc700bce7/istio-proxy/0.log"
Oct 30 19:47:09.536984 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:09.536939    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-bg5dz_5581ec34-b5b8-4c11-a8ae-f31fe0e894f4/istio-proxy/0.log"
Oct 30 19:47:10.412693 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:10.412656    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-hr2xj_a792ece3-61ba-44d8-96e6-b0973bdd83d2/istio-proxy/0.log"
Oct 30 19:47:10.638413 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:10.638387    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-qz2wk_a5e960a4-b8fb-4bf9-a6e6-e8f02b3aa42d/istio-proxy/0.log"
Oct 30 19:47:11.819025 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:11.818985    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:15.414216 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:15.414179    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-658c459c99-dk64c_ae793fdf-7781-4e49-ad45-5bec61829d41/manager/0.log"
Oct 30 19:47:15.426241 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:15.426210    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:15.924319 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:15.924282    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:16.022965 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:16.022940    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-69b7bb8c5c-5b5dp_4b0542b1-baff-4f8d-803a-a7760bc028cd/manager/0.log"
Oct 30 19:47:19.874778 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:47:19.874737    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:47:20.774806 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:20.774772    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:21.029689 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.029623    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5ndc8/must-gather-72hpg"]
Oct 30 19:47:21.030135 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.029944    4853 cpu_manager.go:414] "containerMap: removing container" podUID="5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" containerName="main"
Oct 30 19:47:21.030135 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.029959    4853 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" containerName="main"
Oct 30 19:47:21.030135 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.030047    4853 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5f87fc-76fc-4935-a6d5-8a36fef61d0a" containerName="main"
Oct 30 19:47:21.032882 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.032863    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ndc8/must-gather-72hpg"
Oct 30 19:47:21.087182 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.087159    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ndc8/must-gather-72hpg"]
Oct 30 19:47:21.091332 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.091307    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:21.120835 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.120804    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h27fx\" (UniqueName: \"kubernetes.io/projected/61600de8-9020-4c37-9541-13822d3bd070-kube-api-access-h27fx\") pod \"must-gather-72hpg\" (UID: \"61600de8-9020-4c37-9541-13822d3bd070\") " pod="openshift-must-gather-5ndc8/must-gather-72hpg"
Oct 30 19:47:21.120960 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.120847    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/61600de8-9020-4c37-9541-13822d3bd070-must-gather-output\") pod \"must-gather-72hpg\" (UID: \"61600de8-9020-4c37-9541-13822d3bd070\") " pod="openshift-must-gather-5ndc8/must-gather-72hpg"
Oct 30 19:47:21.131764 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.131744    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5ndc8\"/\"openshift-service-ca.crt\""
Oct 30 19:47:21.131870 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.131768    4853 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5ndc8\"/\"default-dockercfg-xx74q\""
Oct 30 19:47:21.131870 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.131822    4853 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5ndc8\"/\"kube-root-ca.crt\""
Oct 30 19:47:21.221634 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.221614    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h27fx\" (UniqueName: \"kubernetes.io/projected/61600de8-9020-4c37-9541-13822d3bd070-kube-api-access-h27fx\") pod \"must-gather-72hpg\" (UID: \"61600de8-9020-4c37-9541-13822d3bd070\") " pod="openshift-must-gather-5ndc8/must-gather-72hpg"
Oct 30 19:47:21.221746 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.221651    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/61600de8-9020-4c37-9541-13822d3bd070-must-gather-output\") pod \"must-gather-72hpg\" (UID: \"61600de8-9020-4c37-9541-13822d3bd070\") " pod="openshift-must-gather-5ndc8/must-gather-72hpg"
Oct 30 19:47:21.221939 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.221923    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/61600de8-9020-4c37-9541-13822d3bd070-must-gather-output\") pod \"must-gather-72hpg\" (UID: \"61600de8-9020-4c37-9541-13822d3bd070\") " pod="openshift-must-gather-5ndc8/must-gather-72hpg"
Oct 30 19:47:21.277081 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.277056    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h27fx\" (UniqueName: \"kubernetes.io/projected/61600de8-9020-4c37-9541-13822d3bd070-kube-api-access-h27fx\") pod \"must-gather-72hpg\" (UID: \"61600de8-9020-4c37-9541-13822d3bd070\") " pod="openshift-must-gather-5ndc8/must-gather-72hpg"
Oct 30 19:47:21.304931 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.304870    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:21.346553 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.346520    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ndc8/must-gather-72hpg"
Oct 30 19:47:21.726352 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.726313    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ndc8/must-gather-72hpg"]
Oct 30 19:47:21.732628 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:47:21.732601    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61600de8_9020_4c37_9541_13822d3bd070.slice/crio-e2d0b0f35a7a9a6cbd79afd1d126d1bcbd73a8387670267fd0176cd6ad140276 WatchSource:0}: Error finding container e2d0b0f35a7a9a6cbd79afd1d126d1bcbd73a8387670267fd0176cd6ad140276: Status 404 returned error can't find the container with id e2d0b0f35a7a9a6cbd79afd1d126d1bcbd73a8387670267fd0176cd6ad140276
Oct 30 19:47:21.734408 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:21.734390    4853 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:47:22.563905 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:22.563874    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ndc8/must-gather-72hpg" event={"ID":"61600de8-9020-4c37-9541-13822d3bd070","Type":"ContainerStarted","Data":"e2d0b0f35a7a9a6cbd79afd1d126d1bcbd73a8387670267fd0176cd6ad140276"}
Oct 30 19:47:23.570499 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:23.570463    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ndc8/must-gather-72hpg" event={"ID":"61600de8-9020-4c37-9541-13822d3bd070","Type":"ContainerStarted","Data":"3abcfd30a151f88565d5c029b449bc0ed175cb43eefbc360b76b071716573dc7"}
Oct 30 19:47:23.570915 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:23.570515    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ndc8/must-gather-72hpg" event={"ID":"61600de8-9020-4c37-9541-13822d3bd070","Type":"ContainerStarted","Data":"4e3fb921cd244a7ed9c68b3353e2ec4ebf9ce2835cd123572408258860692448"}
Oct 30 19:47:23.682884 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:23.682219    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5ndc8/must-gather-72hpg" podStartSLOduration=2.509740203 podStartE2EDuration="3.682201715s" totalImagesPullingTime="1.172461512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:23.680415885 +0000 UTC m=+4149.850772733" watchObservedRunningTime="2026-10-30 19:47:23.682201715 +0000 UTC m=+4149.852558564"
Oct 30 19:47:27.083382 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:27.083343    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:27.314547 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:27.314485    4853 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:30.928122 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:30.928092    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mqdgz_b81698fa-f5d9-48b0-85fa-29e71a85c441/global-pull-secret-syncer/0.log"
Oct 30 19:47:33.367549 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:33.366943    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-25n6m_47d77dc5-6269-4949-b4fa-60c617073f7c/konnectivity-agent/0.log"
Oct 30 19:47:35.187843 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:35.186920    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-31.us-west-2.compute.internal_38d71fab1c4baec8ab7a9648db13d15f/haproxy/0.log"
Oct 30 19:47:41.408403 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:41.408370    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-658c459c99-dk64c_ae793fdf-7781-4e49-ad45-5bec61829d41/manager/0.log"
Oct 30 19:47:42.306021 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.305981    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-69b7bb8c5c-5b5dp_4b0542b1-baff-4f8d-803a-a7760bc028cd/manager/0.log"
Oct 30 19:47:42.495063 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.495024    4853 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"]
Oct 30 19:47:42.499000 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.498980    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.527857 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.527828    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxx6\" (UniqueName: \"kubernetes.io/projected/3221529f-fc87-48ab-ba0e-7fc527746604-kube-api-access-bnxx6\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.527998 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.527950    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-lib-modules\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.527998 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.527985    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-proc\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.528094 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.528008    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-sys\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.528094 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.528028    4853 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-podres\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.579005 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.578977    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"]
Oct 30 19:47:42.628890 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.628869    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-lib-modules\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.629036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.628908    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-proc\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.629036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.628927    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-sys\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.629036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.628943    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-podres\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.629036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.628965    4853 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxx6\" (UniqueName: \"kubernetes.io/projected/3221529f-fc87-48ab-ba0e-7fc527746604-kube-api-access-bnxx6\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.629036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.629014    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-proc\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.629036 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.629015    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-lib-modules\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.629266 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.629019    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-sys\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.629266 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.629049    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3221529f-fc87-48ab-ba0e-7fc527746604-podres\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.685012 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.684989    4853 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxx6\" (UniqueName: \"kubernetes.io/projected/3221529f-fc87-48ab-ba0e-7fc527746604-kube-api-access-bnxx6\") pod \"perf-node-gather-daemonset-8tbkc\" (UID: \"3221529f-fc87-48ab-ba0e-7fc527746604\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:42.811339 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:42.811310    4853 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:43.180323 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:43.180241    4853 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"]
Oct 30 19:47:43.186849 ip-10-0-129-31 kubenswrapper[4853]: W1030 19:47:43.186806    4853 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3221529f_fc87_48ab_ba0e_7fc527746604.slice/crio-790f6a9a2743be1b02e030dd24a5fd61ac2d153ae17f85bf664cb423f00e7011 WatchSource:0}: Error finding container 790f6a9a2743be1b02e030dd24a5fd61ac2d153ae17f85bf664cb423f00e7011: Status 404 returned error can't find the container with id 790f6a9a2743be1b02e030dd24a5fd61ac2d153ae17f85bf664cb423f00e7011
Oct 30 19:47:43.734751 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:43.734718    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc" event={"ID":"3221529f-fc87-48ab-ba0e-7fc527746604","Type":"ContainerStarted","Data":"56e66cdf7b8d643e1f1ff5b82779cdc15aee76486b66b0b250236b86d21d119d"}
Oct 30 19:47:43.735179 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:43.734767    4853 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc" event={"ID":"3221529f-fc87-48ab-ba0e-7fc527746604","Type":"ContainerStarted","Data":"790f6a9a2743be1b02e030dd24a5fd61ac2d153ae17f85bf664cb423f00e7011"}
Oct 30 19:47:43.735179 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:43.734810    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:43.844884 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:43.844822    4853 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc" podStartSLOduration=1.844808031 podStartE2EDuration="1.844808031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:43.844352131 +0000 UTC m=+4170.014708980" watchObservedRunningTime="2026-10-30 19:47:43.844808031 +0000 UTC m=+4170.015164880"
Oct 30 19:47:49.749579 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:49.749548    4853 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-8tbkc"
Oct 30 19:47:49.873703 ip-10-0-129-31 kubenswrapper[4853]: E1030 19:47:49.873666    4853 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bjg6d" containerName="extractor"
Oct 30 19:47:49.881453 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:49.881421    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bjkm5_2bd81706-d26f-420f-a97d-270ebbc96057/node-exporter/0.log"
Oct 30 19:47:50.292514 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:50.292482    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bjkm5_2bd81706-d26f-420f-a97d-270ebbc96057/kube-rbac-proxy/0.log"
Oct 30 19:47:50.696373 ip-10-0-129-31 kubenswrapper[4853]: I1030 19:47:50.696343    4853 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bjkm5_2bd81706-d26f-420f-a97d-270ebbc96057/init-textfile/0.log"