Oct 31 00:28:05.560315 ip-10-0-132-125 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:28:05.920436 ip-10-0-132-125 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:05.920436 ip-10-0-132-125 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:28:05.920436 ip-10-0-132-125 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:05.920436 ip-10-0-132-125 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:05.925764 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925693    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:05.925764 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925764    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:05.925764 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925768    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925771    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925774    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925776    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925779    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925781    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925784    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925786    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925789    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925791    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925794    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925797    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925800    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925802    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925805    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925807    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925810    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925812    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925815    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925817    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:05.925865 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925820    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925823    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925825    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925828    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925830    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925833    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925835    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925838    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925840    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925843    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925845    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925848    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925850    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925853    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925856    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925859    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925861    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925864    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925867    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925869    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:05.926327 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925872    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925874    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925877    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925879    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925882    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925887    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925891    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925894    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925897    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925899    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925902    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925905    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925907    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925909    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925912    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925915    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925917    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925920    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925922    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:05.926779 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925925    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925928    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925931    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925935    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925938    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925942    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925944    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925948    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925951    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925953    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925955    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925958    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925960    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925963    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925965    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925968    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925971    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925973    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925976    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:05.927224 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925979    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925982    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925984    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925987    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925989    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925992    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925994    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925997    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.925999    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926002    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926004    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926007    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926009    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926013    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926015    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926018    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926021    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926023    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926027    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926030    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:05.927657 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926047    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926050    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926053    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926055    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926058    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926061    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926835    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926842    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926847    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926850    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926853    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926856    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926859    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926862    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926865    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926867    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926870    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926872    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926875    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:05.928127 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926877    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926879    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926882    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926884    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926887    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926889    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926893    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926895    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926898    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926900    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926903    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926906    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926908    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926911    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926913    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926916    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926918    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926921    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926923    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:05.928610 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926926    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926928    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926931    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926933    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926935    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926938    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926940    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926947    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926950    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926952    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926954    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926957    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926959    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926961    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926964    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926966    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926970    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926973    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926976    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926979    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:05.929300 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926981    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926984    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926986    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926989    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926992    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926994    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.926998    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927000    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927003    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927005    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927007    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927010    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927012    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927015    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927018    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927020    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927023    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927025    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927027    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927030    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:05.930006 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927048    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927051    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927053    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927055    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927058    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927060    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927063    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927065    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927068    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927070    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927073    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927075    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927078    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927081    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927085    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927089    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927092    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927095    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927098    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:05.930854 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927100    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927103    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927106    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927109    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927112    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927114    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927116    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927119    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927122    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927124    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927126    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927128    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927131    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927133    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927136    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927612    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927618    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927621    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927623    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927626    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:05.931705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927628    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927631    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927634    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927637    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927640    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927642    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927645    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927648    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927650    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927653    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927656    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927658    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927661    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927663    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927666    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927668    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927671    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927673    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927676    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927678    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:05.932305 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927681    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927683    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927686    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927688    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927691    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927693    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927696    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927698    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927701    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927703    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927705    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927708    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927710    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927718    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927720    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927723    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927726    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927728    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927732    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:05.936811 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927735    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927738    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927741    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927744    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927747    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927750    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927752    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927754    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927757    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927759    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927761    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927764    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927767    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927769    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927772    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927776    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927779    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927782    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927784    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:05.937363 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927787    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927789    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927792    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927795    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927797    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927799    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927802    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927807    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927809    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927812    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927814    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927817    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927819    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927822    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927824    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927827    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927829    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927831    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927834    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927836    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:05.937976 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927839    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927841    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927844    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927846    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927849    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927851    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927854    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927856    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927859    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927861    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927864    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927866    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927868    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927871    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927874    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927877    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927879    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927881    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927884    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927887    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:05.938588 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927890    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927892    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.927895    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928567    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928581    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928588    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928592    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928596    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928600    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928604    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928608    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928612    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928615    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928618    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928622    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928625    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928628    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928631    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928634    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928636    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928639    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928643    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928646    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:28:05.939189 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928649    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928652    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928655    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928659    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928696    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928724    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928727    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928731    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928734    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928737    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928741    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.928744    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929701    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929739    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929897    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929904    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929911    2577 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929916    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929928    2577 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929934    2577 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929939    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929944    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929949    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929957    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:28:05.939797 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929962    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929967    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929974    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929978    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929983    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929988    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929992    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.929997    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930001    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930006    2577 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930012    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930016    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930023    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930028    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930054    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930059    2577 flags.go:64] FLAG: --help="false"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930064    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930069    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930074    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930079    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930087    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930093    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930100    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:28:05.940487 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930105    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930110    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930115    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930119    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930124    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930128    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930133    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930138    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930143    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930147    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930152    2577 flags.go:64] FLAG: --lock-file=""
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930156    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930161    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930166    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930174    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930179    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930183    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930188    2577 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930192    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930198    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930202    2577 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930206    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930214    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930221    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930228    2577 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:28:05.941026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930233    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930237    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930242    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930247    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930251    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930256    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930262    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930275    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930280    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930285    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930289    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930293    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930298    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930302    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930307    2577 flags.go:64] FLAG: --port="10250"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930312    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930316    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0eaf5b5e530924e56"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930321    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930326    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930331    2577 flags.go:64] FLAG: --register-node="true"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930335    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930341    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930346    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930351    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930355    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930361    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:28:05.941644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930366    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930371    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930375    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930380    2577 flags.go:64] FLAG: --runonce="false"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930386    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930391    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930395    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930400    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930404    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930409    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930414    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930418    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930424    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930430    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930434    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930439    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930444    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930448    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930452    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930462    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930467    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930471    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930478    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930482    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930487    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:28:05.942248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930491    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930496    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930500    2577 flags.go:64] FLAG: --v="2"
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930507    2577 flags.go:64] FLAG: --version="false"
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930514    2577 flags.go:64] FLAG: --vmodule=""
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930520    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.930525    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930735    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930744    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930752    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930758    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930765    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930769    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930773    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930777    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930782    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930786    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930791    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930796    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930801    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930805    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:05.942848 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930809    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930814    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930818    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930822    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930826    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930830    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930834    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930838    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930842    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930848    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930852    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930856    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930860    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930863    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930867    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930871    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930875    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930879    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930883    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:05.943352 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930888    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930894    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930900    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930906    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930911    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930915    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930919    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930923    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930927    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930931    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930936    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930942    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930946    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930950    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930954    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930958    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930962    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930966    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930970    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:05.943781 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930975    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930979    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930983    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930987    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930991    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.930996    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931000    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931004    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931008    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931012    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931016    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931020    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931024    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931028    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931048    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931053    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931059    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931064    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931069    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931072    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:05.944243 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931077    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931081    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931085    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931089    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931095    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931099    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931103    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931107    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931111    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931115    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931119    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931123    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931127    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931131    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931136    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931140    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931144    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931151    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931155    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931159    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:05.944705 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931163    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931167    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931171    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931175    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931179    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931183    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931187    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931192    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931197    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931201    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931206    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931210    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931214    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931218    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.931263    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931320    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:05.945296 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931327    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931360    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931364    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931369    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931372    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931376    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931381    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931385    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931389    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931393    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931398    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931402    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931408    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931413    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931419    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931424    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931428    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931433    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931438    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:05.945665 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931442    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931446    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931449    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931454    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931458    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931463    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931468    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931472    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931476    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931480    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931484    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931488    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931492    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931495    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931501    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931505    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931509    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931513    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931517    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:05.946199 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931521    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931525    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931529    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931533    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931537    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931541    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931545    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931549    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931553    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931558    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931562    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931566    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931570    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931574    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931578    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931582    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931585    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931589    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931595    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931599    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:05.946648 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931603    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931607    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931611    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931615    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931620    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931624    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931629    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931635    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931639    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931644    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931648    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931652    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931656    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931660    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931664    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931668    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931672    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931676    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931680    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931684    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:05.947113 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931688    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931692    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931697    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931701    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931705    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931709    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931712    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931719    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931725    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931730    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931736    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931741    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931745    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931749    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931754    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931758    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931764    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931768    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931772    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:05.947573 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931778    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931783    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931787    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931791    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931795    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931799    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931803    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.931807    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.941865    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.941878    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.941891    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.941895    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941948    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941954    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941957    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941961    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941964    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941967    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941969    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:05.948056 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941973    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941975    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941978    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941980    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941983    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941985    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941988    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941990    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941992    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941995    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.941997    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942000    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942002    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942005    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942007    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942009    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942012    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942015    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942017    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942020    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:05.948496 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942022    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942025    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942028    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942031    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942050    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942053    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942055    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942058    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942060    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942063    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942066    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942068    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942071    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942074    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942077    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942079    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942081    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942084    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942086    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942089    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:05.948943 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942091    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942094    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942096    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942098    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942101    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942103    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942105    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942108    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942110    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942113    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942115    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942117    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942120    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942123    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942125    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942128    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942130    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942133    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942135    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:05.949414 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942138    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942140    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942142    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942145    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942148    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942150    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942152    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942155    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942157    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942160    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942162    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942164    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942166    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942169    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942171    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942174    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942176    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942178    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942180    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942183    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:05.949843 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942185    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942188    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942190    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942192    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942195    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942197    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942200    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942202    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942206    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942210    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942213    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942216    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942218    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942222    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942226    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942230    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942233    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942235    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:05.950318 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942238    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942240    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.942267    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942401    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942405    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942409    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942411    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942414    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942416    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942419    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942421    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942424    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942426    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942429    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942431    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942433    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:05.950754 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942436    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942439    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942441    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942444    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942446    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942450    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942453    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942456    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942459    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942461    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942464    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942466    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942469    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942471    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942473    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942476    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942479    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942481    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942483    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:05.951205 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942486    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942488    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942490    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942493    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942496    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942498    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942501    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942503    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942505    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942508    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942510    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942513    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942515    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942517    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942520    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942522    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942525    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942527    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942530    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942532    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:05.951635 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942535    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942537    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942540    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942542    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942544    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942547    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942549    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942551    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942555    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942559    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942562    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942564    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942567    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942569    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942572    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942574    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942576    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942579    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942581    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942583    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:05.952094 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942586    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942588    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942591    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942594    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942596    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942599    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942601    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942604    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942606    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942609    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942611    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942613    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942616    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942618    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942620    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942623    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942625    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942627    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942630    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942632    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:05.952551 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942635    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942637    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942640    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942642    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942644    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942647    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942649    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942651    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942654    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942656    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942658    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942661    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942663    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942668    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.942690    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:05.953002 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942720    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942724    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942727    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942730    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942732    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942735    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942737    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942740    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942743    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942745    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942747    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942750    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942753    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942755    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942758    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942760    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942763    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942765    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942768    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942770    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:05.953374 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942773    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942776    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942778    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942781    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942783    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942785    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942788    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942790    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942792    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942795    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942797    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942801    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942804    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942806    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942809    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942811    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942813    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942816    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942818    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942821    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:05.953823 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942823    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942826    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942828    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942830    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942833    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942835    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942837    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942840    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942842    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942845    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942847    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942850    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942852    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942856    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942859    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942862    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942866    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942870    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942873    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:05.954303 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942876    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942878    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942881    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942883    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942903    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942908    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942911    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942913    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942916    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942918    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942921    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942923    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942925    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942928    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942930    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942933    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942935    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942937    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942939    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942942    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:05.954725 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942945    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942947    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942949    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942951    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942954    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942956    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942960    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942963    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942965    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942968    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942970    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942973    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942975    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942977    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942980    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942982    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942985    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942987    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942990    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:05.955251 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942995    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.942998    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.943001    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.943003    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.943006    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.943008    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.943011    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:05.943013    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.943071    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.946351    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.947175    2577 server.go:1014] "Starting client certificate rotation"
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.947269    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.947303    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.949048    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:28:05.955708 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.950785    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:28:05.961923 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.961901    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:28:05.965556 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.965537    2577 log.go:25] "Validated CRI v1 image API"
Oct 31 00:28:05.966318 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.966296    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:28:05.968742 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.968723    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 90e9edcb-ed98-4137-b600-e2f6ad30a1eb:/dev/nvme0n1p4 fc2ad222-b3fa-420f-8478-5423111f0e52:/dev/nvme0n1p3]
Oct 31 00:28:05.968801 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.968742    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 00:28:05.973549 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.973445    2577 manager.go:217] Machine: {Timestamp:2026-10-31 00:28:05.972148626 +0000 UTC m=+0.308403811 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200148 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22b9b7f62885fbdcc758ad0b7e8384 SystemUUID:ec22b9b7-f628-85fb-dcc7-58ad0b7e8384 BootID:81186f95-cfea-4de4-816b-d40bdc47eced Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:cc:10:c9:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:cc:10:c9:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:25:92:a3:5c:77 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:28:05.973549 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.973548    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:28:05.973654 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.973642    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:28:05.975390 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.975371    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:28:05.975512 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.975392    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-125.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:28:05.976099 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.976090    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:28:05.976134 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.976101    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:28:05.976134 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.976129    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:28:05.976186 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.976141    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:28:05.977203 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.977193    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:28:05.981328 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.981316    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:28:05.981441 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.981432    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:28:05.982216 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.982206    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:28:05.982253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.982221    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:28:05.982253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.982231    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:28:05.982253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.982240    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:28:05.982253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.982247    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:28:05.982532 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.982522    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:28:05.984877 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.984860    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:28:05.991720 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.991708    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:28:05.992645 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.992633    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:28:05.992679 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.992662    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:28:05.992805 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.992796    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:28:05.992838 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.992811    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:28:05.992838 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.992817    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:28:05.992838 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.992823    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:28:05.992838 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.992828    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:28:05.992969 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.992840    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:28:05.993873 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.993865    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:28:05.993903 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.993874    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:28:05.993903 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.993880    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:28:05.993903 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.993886    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:28:05.993903 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.993898    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:28:05.994017 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.993907    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:28:05.994912 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.994904    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:28:05.994946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.994913    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:28:05.998692 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.998673    2577 server.go:1263] "Started kubelet"
Oct 31 00:28:05.998784 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.998754    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:28:05.998867 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.998823    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:28:05.998897 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:05.998886    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:28:06.000330 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.000305    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:28:06.000978 ip-10-0-132-125 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:28:06.003211 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.003195    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:28:06.006341 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.006322    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 00:28:06.008554 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.008536    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:28:06.008554 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.008544    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:28:06.008917 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.008892    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:06.009245 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.009222    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:28:06.009367 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.009269    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:28:06.009402 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.009372    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:28:06.009492 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.009479    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:28:06.009521 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.009495    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:28:06.009903 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.009890    2577 factory.go:55] Registering systemd factory
Oct 31 00:28:06.009960 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.009905    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:28:06.010101 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.010086    2577 factory.go:153] Registering CRI-O factory
Oct 31 00:28:06.010157 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.010104    2577 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:28:06.010157 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.010151    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:28:06.010227 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.010194    2577 factory.go:103] Registering Raw factory
Oct 31 00:28:06.010227 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.010206    2577 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:28:06.011218 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.011204    2577 manager.go:319] Starting recovery of all containers
Oct 31 00:28:06.020428 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.020416    2577 manager.go:324] Recovery completed
Oct 31 00:28:06.024478 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.024467    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:06.026415 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.026376    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:06.026482 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.026431    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:06.026482 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.026444    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:06.026893 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.026880    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:28:06.026893 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.026890    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:06.027005 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.026891    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:28:06.027005 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.026925    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:28:06.027960 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.027937    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:06.028030 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.027977    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:06.028030 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.027991    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:06.029250 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.029234    2577 policy_none.go:50] "Start"
Oct 31 00:28:06.029250 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.029252    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:28:06.029356 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.029267    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:28:06.031250 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.031239    2577 policy_none.go:44] "Start"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.085693    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.086571    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.087023    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.087057    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.087096    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.087129    2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.089452    2577 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.089490    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.089503    2577 server.go:87] "Starting device plugin registration server"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.089744    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.089758    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.089832    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.089904    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.089911    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.090467    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:28:06.095936 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.090517    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:06.136780 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.136748    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-125.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 00:28:06.136780 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.136759    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-125.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:28:06.137925 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.136760    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-125.us-west-2.compute.internal.18e375b277d4bb7c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-125.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-125.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:05.998648188 +0000 UTC m=+0.334903373,LastTimestamp:2026-10-31 00:28:05.998648188 +0000 UTC m=+0.334903373,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-125.us-west-2.compute.internal,}"
Oct 31 00:28:06.187340 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.187293    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"]
Oct 31 00:28:06.187409 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.187344    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:06.188134 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.188119    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:06.188185 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.188146    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:06.188185 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.188156    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:06.190292 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.190277    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:06.190329 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.190294    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:06.190403 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.190392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.190440 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.190415    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:06.190972 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.190953    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:06.190972 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.190966    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:06.191167 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.190988    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:06.191167 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.190958    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:06.191167 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.190989    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:06.191167 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.191012    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:06.191167 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.191000    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:06.191167 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.191049    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:06.191167 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.191052    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.191167 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.191064    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:06.193008 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.192992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.193089 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.193016    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:06.193684 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.193668    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:06.193771 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.193700    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:06.193771 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.193711    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:06.214192 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.214173    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:28:06.214292 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.214225    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dtxp6"
Oct 31 00:28:06.219331 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.219318    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found" node="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.221629 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.221614    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found" node="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.260314 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.260292    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-125.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 00:28:06.260429 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.260323    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 00:28:06.260484 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.260433    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 31 00:28:06.260650 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.260570    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-125.us-west-2.compute.internal.18e375b2797c67f9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-125.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-125.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-125.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:06.026414073 +0000 UTC m=+0.362669262,LastTimestamp:2026-10-31 00:28:06.026414073 +0000 UTC m=+0.362669262,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-125.us-west-2.compute.internal,}"
Oct 31 00:28:06.310500 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.310478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/859c761ccba0302346f6d0c201b0e803-config\") pod \"kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"859c761ccba0302346f6d0c201b0e803\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.310569 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.310503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.310569 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.310518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.343495 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.343475    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:06.344935 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.344920    2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-125.us-west-2.compute.internal\" not found" node="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.411083 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.411058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/859c761ccba0302346f6d0c201b0e803-config\") pod \"kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"859c761ccba0302346f6d0c201b0e803\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.411180 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.411086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.411180 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.411101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.411180 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.411146    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.411180 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.411158    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/859c761ccba0302346f6d0c201b0e803-config\") pod \"kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"859c761ccba0302346f6d0c201b0e803\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.411180 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.411164    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.437624 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.437584    2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-125.us-west-2.compute.internal\" not found" node="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.524335 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.524314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.527480 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.527465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.545067 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.545030    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:06.545773 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.545759    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:06.545838 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.545786    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:06.545838 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.545796    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:06.545838 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.545819    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.601571 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.601555    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:06.601655 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.601578    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-125.us-west-2.compute.internal\": node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:06.711388 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.711324    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:06.811809 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.811778    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:06.912313 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.912281    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:06.947072 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.947012    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 00:28:06.947465 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:06.947218    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 00:28:06.947465 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:06.947193    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://abfbb1200564c4aa4933eb32a0e1d50d-bea5dad701e52471.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-132-125.us-west-2.compute.internal.18e375b2797c67f9\": read tcp 10.0.132.125:39386->77.112.219.56:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-132-125.us-west-2.compute.internal.18e375b2797c67f9  default   4648 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-125.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-125.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-125.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:06 +0000 UTC,LastTimestamp:2026-10-31 00:28:06.190976313 +0000 UTC m=+0.527231498,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-125.us-west-2.compute.internal,}"
Oct 31 00:28:07.012560 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:07.012525    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:07.113012 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:07.112990    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:07.190860 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.190836    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:07.191665 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.191649    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:07.191754 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.191681    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:07.191754 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.191696    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:07.213452 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:07.213432    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:07.220666 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.220640    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:23:06 +0000 UTC" deadline="2028-07-14 06:36:04.63088353 +0000 UTC"
Oct 31 00:28:07.220666 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.220663    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14934h7m57.41022308s"
Oct 31 00:28:07.314458 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:07.314394    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:07.354893 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.354872    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:07.409098 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.409072    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:07.476677 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.476651    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:07.479427 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.479414    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:07.539907 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.539882    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:07.737513 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.737483    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:07.737854 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.737835    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:07.810680 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:07.810645    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4919af72af1b023905c8a7373620fc2c.slice/crio-3d4dcfbc57887f3ae26e161794606c8a4eec7b2c4a0a5c246c7b3425f645a1fa WatchSource:0}: Error finding container 3d4dcfbc57887f3ae26e161794606c8a4eec7b2c4a0a5c246c7b3425f645a1fa: Status 404 returned error can't find the container with id 3d4dcfbc57887f3ae26e161794606c8a4eec7b2c4a0a5c246c7b3425f645a1fa
Oct 31 00:28:07.815355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.815339    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:28:07.984897 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:07.984812    2577 apiserver.go:52] "Watching apiserver"
Oct 31 00:28:08.010940 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.010908    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 00:28:08.092020 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.091959    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal" event={"ID":"859c761ccba0302346f6d0c201b0e803","Type":"ContainerStarted","Data":"c4b98ab84513804906c41697e62ab0a3f72ac8f3870f11868c9f05645281992b"}
Oct 31 00:28:08.093140 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.093119    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal" event={"ID":"4919af72af1b023905c8a7373620fc2c","Type":"ContainerStarted","Data":"3d4dcfbc57887f3ae26e161794606c8a4eec7b2c4a0a5c246c7b3425f645a1fa"}
Oct 31 00:28:08.101577 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.101553    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:28:08.101915 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.101897    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-z98bg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s","openshift-image-registry/node-ca-fvksv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal","openshift-multus/multus-76vsc","openshift-multus/network-metrics-daemon-7smc9","openshift-network-diagnostics/network-check-target-pdpbv","kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-h8q7j","kube-system/global-pull-secret-syncer-wdzhr","openshift-cluster-node-tuning-operator/tuned-sxssn","kube-system/konnectivity-agent-l64l2","openshift-dns/node-resolver-29ctl","openshift-network-operator/iptables-alerter-8p6vl"]
Oct 31 00:28:08.106107 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.106087    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:08.106197 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.106154    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:08.108267 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.108254    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:08.110479 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.110462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:08.112764 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.112744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:08.115000 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.114957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.117611 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.117589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-kubelet-config\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:08.117693 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.117625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-dbus\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:08.117693 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.117690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/693a3c1f-8228-4691-bd99-f05fdf784cd4-hosts-file\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:08.117814 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.117711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693a3c1f-8228-4691-bd99-f05fdf784cd4-tmp-dir\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:08.117814 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.117733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tz7l\" (UniqueName: \"kubernetes.io/projected/693a3c1f-8228-4691-bd99-f05fdf784cd4-kube-api-access-5tz7l\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:08.117814 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.117789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:08.117907 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.117853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5703958-5751-4c4d-b979-01585c84cd46-host\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:08.117942 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.117909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5703958-5751-4c4d-b979-01585c84cd46-serviceca\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:08.117977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.117939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvlx\" (UniqueName: \"kubernetes.io/projected/f5703958-5751-4c4d-b979-01585c84cd46-kube-api-access-dnvlx\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:08.118424 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.118406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:08.118501 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.118464    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:08.120039 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.120022    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.122796 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.122778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:08.122859 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.122845    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:08.125274 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.125254    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.127993 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.127974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.130439 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.130423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.132736 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.132718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:08.168538 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.168520    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:28:08.168876 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.168853    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzctz\""
Oct 31 00:28:08.168968 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.168952    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:28:08.169030 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.168975    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:28:08.169128 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.169063    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:28:08.169191 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.169166    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4l8km\""
Oct 31 00:28:08.169256 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.169238    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:28:08.171768 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.171751    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:08.171832 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.171751    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:08.171832 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.171779    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-klgwd\""
Oct 31 00:28:08.171832 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.171805    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:28:08.173601 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.173586    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:28:08.173694 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.173680    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5b488\""
Oct 31 00:28:08.173731 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.173723    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:28:08.173765 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.173747    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:28:08.173865 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.173854    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:28:08.178874 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.178859    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t6xpx\""
Oct 31 00:28:08.178970 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.178958    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:28:08.179040 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.179030    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:28:08.183635 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.183621    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:28:08.184070 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.184055    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:28:08.184135 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.184065    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:28:08.211304 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.211286    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:28:08.215531 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.215516    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:28:08.215622 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.215546    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2sfqr\""
Oct 31 00:28:08.218232 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-systemd\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.218309 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.218309 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.218309 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-device-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.218309 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-systemd-units\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.218309 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218294    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:28:08.218490 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-etc-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.218490 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218355    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s5c24\""
Oct 31 00:28:08.218490 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218376    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l8m77\""
Oct 31 00:28:08.218490 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.218490 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-kubelet-config\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:08.218490 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-sys-fs\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.218664 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-hostroot\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.218664 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218525    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-kubelet-config\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:08.218664 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44wtx\" (UniqueName: \"kubernetes.io/projected/c8302492-d696-40ad-bffd-431b35319e22-kube-api-access-44wtx\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.218664 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-systemd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.218664 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-env-overrides\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.218664 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-script-lib\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.218664 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/693a3c1f-8228-4691-bd99-f05fdf784cd4-hosts-file\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:08.218859 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693a3c1f-8228-4691-bd99-f05fdf784cd4-tmp-dir\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:08.218859 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218725    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/693a3c1f-8228-4691-bd99-f05fdf784cd4-hosts-file\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:08.218859 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-conf\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.218859 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-etc-tuned\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.218982 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-socket-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.218982 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-var-lib-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.218982 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-socket-dir-parent\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.218982 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218960    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693a3c1f-8228-4691-bd99-f05fdf784cd4-tmp-dir\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:08.219161 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.218975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-system-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.219161 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-k8s-cni-cncf-io\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.219161 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-cnibin\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.219161 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.219161 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-node-log\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.219161 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-bin\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.219422 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-system-cni-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.219422 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvrb\" (UniqueName: \"kubernetes.io/projected/b1402f00-5adc-44cb-961c-a2556e51529b-kube-api-access-6lvrb\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.219422 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-etc-kubernetes\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.219422 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovn-node-metrics-cert\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.219422 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvlx\" (UniqueName: \"kubernetes.io/projected/f5703958-5751-4c4d-b979-01585c84cd46-kube-api-access-dnvlx\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:08.219422 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cni-binary-copy\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.219707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-daemon-config\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.219707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-kubernetes\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.219707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-tmp\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.219707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-etc-selinux\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.219707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-kubelet\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.219707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-netns\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.219707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-var-lib-kubelet\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.219707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-ovn\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-netns\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-host\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ab38846-5d17-4bf1-92c2-ac823690fc98-konnectivity-ca\") pod \"konnectivity-agent-l64l2\" (UID: \"7ab38846-5d17-4bf1-92c2-ac823690fc98\") " pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-iptables-alerter-script\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-slash\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-multus\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-kubelet\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.219991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysconfig\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.220026 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-sys\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-host-slash\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86wrt\" (UniqueName: \"kubernetes.io/projected/a27a3030-72b2-4223-b95e-c3997a8eb0f6-kube-api-access-86wrt\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-bin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-495pb\" (UniqueName: \"kubernetes.io/projected/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-kube-api-access-495pb\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5703958-5751-4c4d-b979-01585c84cd46-serviceca\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-dbus\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.220229    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-registration-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-log-socket\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.220298    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret podName:261b0474-eb3d-4f3c-b33d-01856306e928 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:08.720270685 +0000 UTC m=+3.056525859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret") pod "global-pull-secret-syncer-wdzhr" (UID: "261b0474-eb3d-4f3c-b33d-01856306e928") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-config\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-conf-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ab38846-5d17-4bf1-92c2-ac823690fc98-agent-certs\") pod \"konnectivity-agent-l64l2\" (UID: \"7ab38846-5d17-4bf1-92c2-ac823690fc98\") " pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:08.220504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5k6g\" (UniqueName: \"kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g\") pod \"network-check-target-pdpbv\" (UID: \"d48c64d8-b46d-48e0-830e-d9ec6a92ff06\") " pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220534    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-dbus\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q9pr\" (UniqueName: \"kubernetes.io/projected/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kube-api-access-2q9pr\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220544    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5703958-5751-4c4d-b979-01585c84cd46-serviceca\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-netd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tz7l\" (UniqueName: \"kubernetes.io/projected/693a3c1f-8228-4691-bd99-f05fdf784cd4-kube-api-access-5tz7l\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cnibin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220775    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-multus-certs\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220776    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bv698\""
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfs9t\" (UniqueName: \"kubernetes.io/projected/d7ef07d0-5483-4176-bfb2-4e03c9be4498-kube-api-access-kfs9t\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f8k\" (UniqueName: \"kubernetes.io/projected/0e67064c-ba5d-4780-9257-9df20f304f29-kube-api-access-r4f8k\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220918    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-modprobe-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220940    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-run\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220976    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220984    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:28:08.220977 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220976    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:28:08.221578 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.220988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-lib-modules\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.221578 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.221010    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:28:08.221578 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.221033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5703958-5751-4c4d-b979-01585c84cd46-host\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:08.221578 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.221079    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5703958-5751-4c4d-b979-01585c84cd46-host\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:08.226138 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.224051    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:28:08.226138 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.224085    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:28:08.226138 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.224146    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-db6mc"
Oct 31 00:28:08.282721 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.282701    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:08.286884 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.286858    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:28:08.290322 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.290302    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tz7l\" (UniqueName: \"kubernetes.io/projected/693a3c1f-8228-4691-bd99-f05fdf784cd4-kube-api-access-5tz7l\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:08.290322 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.290317    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvlx\" (UniqueName: \"kubernetes.io/projected/f5703958-5751-4c4d-b979-01585c84cd46-kube-api-access-dnvlx\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:08.322321 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.322389 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5k6g\" (UniqueName: \"kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g\") pod \"network-check-target-pdpbv\" (UID: \"d48c64d8-b46d-48e0-830e-d9ec6a92ff06\") " pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:08.322389 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9pr\" (UniqueName: \"kubernetes.io/projected/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kube-api-access-2q9pr\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.322389 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-netd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.322389 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cnibin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-multus-certs\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfs9t\" (UniqueName: \"kubernetes.io/projected/d7ef07d0-5483-4176-bfb2-4e03c9be4498-kube-api-access-kfs9t\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322475    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322507    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-multus-certs\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322511    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cnibin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322512    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-netd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f8k\" (UniqueName: \"kubernetes.io/projected/0e67064c-ba5d-4780-9257-9df20f304f29-kube-api-access-r4f8k\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-modprobe-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-run\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-lib-modules\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-systemd\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322705    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-run\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.322734 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322727    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-modprobe-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322751    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-systemd\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-device-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322762    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-lib-modules\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322765    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-systemd-units\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322794    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-device-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-etc-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-sys-fs\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-hostroot\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44wtx\" (UniqueName: \"kubernetes.io/projected/c8302492-d696-40ad-bffd-431b35319e22-kube-api-access-44wtx\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-etc-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322896    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322904    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-hostroot\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322852    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-systemd-units\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-systemd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322923    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-sys-fs\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.323473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322943    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-systemd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-env-overrides\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322949    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.322979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-script-lib\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-conf\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-etc-tuned\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-socket-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-var-lib-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-socket-dir-parent\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323145    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-conf\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323158    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-var-lib-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-system-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-k8s-cni-cncf-io\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323208    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-socket-dir-parent\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323215    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-socket-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323247    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-k8s-cni-cncf-io\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-cnibin\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.324360 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323262    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323268    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-cnibin\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323210    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-system-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-node-log\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-bin\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-system-cni-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323394    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-node-log\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323409    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-bin\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323436    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-system-cni-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323440    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvrb\" (UniqueName: \"kubernetes.io/projected/b1402f00-5adc-44cb-961c-a2556e51529b-kube-api-access-6lvrb\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-etc-kubernetes\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323507    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-env-overrides\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovn-node-metrics-cert\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323554    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-etc-kubernetes\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cni-binary-copy\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.325253 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-daemon-config\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323609    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-script-lib\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-kubernetes\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-tmp\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323676    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-kubernetes\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-etc-selinux\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-kubelet\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-netns\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323758    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-etc-selinux\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-var-lib-kubelet\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323803    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-kubelet\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-ovn\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323850    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-var-lib-kubelet\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-netns\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.326092 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323884    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-netns\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-host\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ab38846-5d17-4bf1-92c2-ac823690fc98-konnectivity-ca\") pod \"konnectivity-agent-l64l2\" (UID: \"7ab38846-5d17-4bf1-92c2-ac823690fc98\") " pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-iptables-alerter-script\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-slash\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.323986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-multus\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-kubelet\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysconfig\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324029    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324070    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-slash\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324089    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-multus\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324085    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-ovn\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-sys\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-host-slash\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324149    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-kubelet\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86wrt\" (UniqueName: \"kubernetes.io/projected/a27a3030-72b2-4223-b95e-c3997a8eb0f6-kube-api-access-86wrt\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324200    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-host\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.326687 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324198    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324117    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-netns\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324267    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-bin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324353    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysconfig\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495pb\" (UniqueName: \"kubernetes.io/projected/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-kube-api-access-495pb\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-registration-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-log-socket\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-config\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-conf-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ab38846-5d17-4bf1-92c2-ac823690fc98-agent-certs\") pod \"konnectivity-agent-l64l2\" (UID: \"7ab38846-5d17-4bf1-92c2-ac823690fc98\") " pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324586    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-host-slash\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324657    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-bin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324314    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-sys\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324872    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cni-binary-copy\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324929    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-conf-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324936    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-daemon-config\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.327355 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.324973    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-registration-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.327941 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.325012    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:08.327941 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.325016    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-log-socket\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.327941 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.325079    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs podName:0e67064c-ba5d-4780-9257-9df20f304f29 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:08.825063236 +0000 UTC m=+3.161318413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs") pod "network-metrics-daemon-7smc9" (UID: "0e67064c-ba5d-4780-9257-9df20f304f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:08.327941 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.325164    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-config\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.327941 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.325507    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.327941 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.325703    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ab38846-5d17-4bf1-92c2-ac823690fc98-konnectivity-ca\") pod \"konnectivity-agent-l64l2\" (UID: \"7ab38846-5d17-4bf1-92c2-ac823690fc98\") " pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:08.327941 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.326090    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-etc-tuned\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.327941 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.326148    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovn-node-metrics-cert\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.327941 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.326170    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-iptables-alerter-script\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:08.327941 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.326375    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-tmp\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.327941 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.327350    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ab38846-5d17-4bf1-92c2-ac823690fc98-agent-certs\") pod \"konnectivity-agent-l64l2\" (UID: \"7ab38846-5d17-4bf1-92c2-ac823690fc98\") " pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:08.394583 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.394557    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:08.394583 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.394575    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:08.394583 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.394584    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l5k6g for pod openshift-network-diagnostics/network-check-target-pdpbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:08.394754 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.394640    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g podName:d48c64d8-b46d-48e0-830e-d9ec6a92ff06 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:08.89462822 +0000 UTC m=+3.230883407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l5k6g" (UniqueName: "kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g") pod "network-check-target-pdpbv" (UID: "d48c64d8-b46d-48e0-830e-d9ec6a92ff06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:08.396921 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.396897    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f8k\" (UniqueName: \"kubernetes.io/projected/0e67064c-ba5d-4780-9257-9df20f304f29-kube-api-access-r4f8k\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:08.397686 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.397665    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44wtx\" (UniqueName: \"kubernetes.io/projected/c8302492-d696-40ad-bffd-431b35319e22-kube-api-access-44wtx\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.398198 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.398179    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfs9t\" (UniqueName: \"kubernetes.io/projected/d7ef07d0-5483-4176-bfb2-4e03c9be4498-kube-api-access-kfs9t\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.398769 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.398749    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-495pb\" (UniqueName: \"kubernetes.io/projected/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-kube-api-access-495pb\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:08.398849 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.398751    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9pr\" (UniqueName: \"kubernetes.io/projected/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kube-api-access-2q9pr\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.398902 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.398885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86wrt\" (UniqueName: \"kubernetes.io/projected/a27a3030-72b2-4223-b95e-c3997a8eb0f6-kube-api-access-86wrt\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.399042 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.399028    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvrb\" (UniqueName: \"kubernetes.io/projected/b1402f00-5adc-44cb-961c-a2556e51529b-kube-api-access-6lvrb\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.422171 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.422155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:08.427560 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.427546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:08.433605 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:08.433581    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693a3c1f_8228_4691_bd99_f05fdf784cd4.slice/crio-f653f495285a89d3f558729b86f578d22e2e440db158445246c22cc032cf67db WatchSource:0}: Error finding container f653f495285a89d3f558729b86f578d22e2e440db158445246c22cc032cf67db: Status 404 returned error can't find the container with id f653f495285a89d3f558729b86f578d22e2e440db158445246c22cc032cf67db
Oct 31 00:28:08.435669 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.435556    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:08.437975 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:08.437952    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5703958_5751_4c4d_b979_01585c84cd46.slice/crio-7d7183f126a97fedb177bb2869ecb7eeebb9b8194d8d3e2c2075bb5d11fe866f WatchSource:0}: Error finding container 7d7183f126a97fedb177bb2869ecb7eeebb9b8194d8d3e2c2075bb5d11fe866f: Status 404 returned error can't find the container with id 7d7183f126a97fedb177bb2869ecb7eeebb9b8194d8d3e2c2075bb5d11fe866f
Oct 31 00:28:08.439888 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.439874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-76vsc"
Oct 31 00:28:08.444412 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:08.444390    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fad5d0_0d3a_4ebe_a07b_1899400059a7.slice/crio-efc2ad77731ea8e8e9d413d9c176a4569eacfea00892764a8acba158ce419e5f WatchSource:0}: Error finding container efc2ad77731ea8e8e9d413d9c176a4569eacfea00892764a8acba158ce419e5f: Status 404 returned error can't find the container with id efc2ad77731ea8e8e9d413d9c176a4569eacfea00892764a8acba158ce419e5f
Oct 31 00:28:08.449537 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:08.449519    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ef07d0_5483_4176_bfb2_4e03c9be4498.slice/crio-91827ae633ef0d4dc544a5124cff2d6c01b8156f9728bea844a8ad5713880a61 WatchSource:0}: Error finding container 91827ae633ef0d4dc544a5124cff2d6c01b8156f9728bea844a8ad5713880a61: Status 404 returned error can't find the container with id 91827ae633ef0d4dc544a5124cff2d6c01b8156f9728bea844a8ad5713880a61
Oct 31 00:28:08.465198 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.465178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:08.467496 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.467479    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:08.468765 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.468746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:08.474458 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.474435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:08.474655 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:08.474636    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d3cbf5_134e_4f86_8b33_04ef38cdb685.slice/crio-5ca98829f18eb3109da7316310c5cdb18d632f9adabb09daa11c186b13077209 WatchSource:0}: Error finding container 5ca98829f18eb3109da7316310c5cdb18d632f9adabb09daa11c186b13077209: Status 404 returned error can't find the container with id 5ca98829f18eb3109da7316310c5cdb18d632f9adabb09daa11c186b13077209
Oct 31 00:28:08.479234 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:08.479202    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1402f00_5adc_44cb_961c_a2556e51529b.slice/crio-dac2a2c016684a2192de8284e955022ba30f23dc6bcfd2e1938cf517b903bf91 WatchSource:0}: Error finding container dac2a2c016684a2192de8284e955022ba30f23dc6bcfd2e1938cf517b903bf91: Status 404 returned error can't find the container with id dac2a2c016684a2192de8284e955022ba30f23dc6bcfd2e1938cf517b903bf91
Oct 31 00:28:08.479883 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:08.479863    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8302492_d696_40ad_bffd_431b35319e22.slice/crio-87d098e65edbb39ef8d0c24afdb590054c01ed748a29b33732b52c7eafa88962 WatchSource:0}: Error finding container 87d098e65edbb39ef8d0c24afdb590054c01ed748a29b33732b52c7eafa88962: Status 404 returned error can't find the container with id 87d098e65edbb39ef8d0c24afdb590054c01ed748a29b33732b52c7eafa88962
Oct 31 00:28:08.484488 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:08.484469    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab38846_5d17_4bf1_92c2_ac823690fc98.slice/crio-4045b6bbec4c2576d027ccc451a90f03228f3a12d53680353956648ad43643c6 WatchSource:0}: Error finding container 4045b6bbec4c2576d027ccc451a90f03228f3a12d53680353956648ad43643c6: Status 404 returned error can't find the container with id 4045b6bbec4c2576d027ccc451a90f03228f3a12d53680353956648ad43643c6
Oct 31 00:28:08.487583 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.487564    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:08.496644 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:08.496626    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27a3030_72b2_4223_b95e_c3997a8eb0f6.slice/crio-59d770ac32b29db330d9d03ff2a4e039a22e00206091186d6c4afbeeefce25ef WatchSource:0}: Error finding container 59d770ac32b29db330d9d03ff2a4e039a22e00206091186d6c4afbeeefce25ef: Status 404 returned error can't find the container with id 59d770ac32b29db330d9d03ff2a4e039a22e00206091186d6c4afbeeefce25ef
Oct 31 00:28:08.726854 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.726771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:08.726969 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.726928    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:08.727004 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.726987    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret podName:261b0474-eb3d-4f3c-b33d-01856306e928 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:09.726973123 +0000 UTC m=+4.063228300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret") pod "global-pull-secret-syncer-wdzhr" (UID: "261b0474-eb3d-4f3c-b33d-01856306e928") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:08.827222 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.827186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:08.827358 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.827301    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:08.827403 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.827360    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs podName:0e67064c-ba5d-4780-9257-9df20f304f29 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:09.827344949 +0000 UTC m=+4.163600122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs") pod "network-metrics-daemon-7smc9" (UID: "0e67064c-ba5d-4780-9257-9df20f304f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:08.927972 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:08.927946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5k6g\" (UniqueName: \"kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g\") pod \"network-check-target-pdpbv\" (UID: \"d48c64d8-b46d-48e0-830e-d9ec6a92ff06\") " pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:08.928131 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.928084    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:08.928131 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.928098    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:08.928131 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.928110    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l5k6g for pod openshift-network-diagnostics/network-check-target-pdpbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:08.928256 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:08.928171    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g podName:d48c64d8-b46d-48e0-830e-d9ec6a92ff06 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:09.928153751 +0000 UTC m=+4.264408928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5k6g" (UniqueName: "kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g") pod "network-check-target-pdpbv" (UID: "d48c64d8-b46d-48e0-830e-d9ec6a92ff06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:09.095898 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.095802    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l64l2" event={"ID":"7ab38846-5d17-4bf1-92c2-ac823690fc98","Type":"ContainerStarted","Data":"4045b6bbec4c2576d027ccc451a90f03228f3a12d53680353956648ad43643c6"}
Oct 31 00:28:09.101322 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.101288    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sxssn" event={"ID":"c8302492-d696-40ad-bffd-431b35319e22","Type":"ContainerStarted","Data":"87d098e65edbb39ef8d0c24afdb590054c01ed748a29b33732b52c7eafa88962"}
Oct 31 00:28:09.102540 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.102515    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8q7j" event={"ID":"b1402f00-5adc-44cb-961c-a2556e51529b","Type":"ContainerStarted","Data":"dac2a2c016684a2192de8284e955022ba30f23dc6bcfd2e1938cf517b903bf91"}
Oct 31 00:28:09.103917 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.103892    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s" event={"ID":"34d3cbf5-134e-4f86-8b33-04ef38cdb685","Type":"ContainerStarted","Data":"5ca98829f18eb3109da7316310c5cdb18d632f9adabb09daa11c186b13077209"}
Oct 31 00:28:09.105497 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.105473    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-76vsc" event={"ID":"d7ef07d0-5483-4176-bfb2-4e03c9be4498","Type":"ContainerStarted","Data":"91827ae633ef0d4dc544a5124cff2d6c01b8156f9728bea844a8ad5713880a61"}
Oct 31 00:28:09.106955 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.106928    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fvksv" event={"ID":"f5703958-5751-4c4d-b979-01585c84cd46","Type":"ContainerStarted","Data":"7d7183f126a97fedb177bb2869ecb7eeebb9b8194d8d3e2c2075bb5d11fe866f"}
Oct 31 00:28:09.108473 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.108448    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg" event={"ID":"a27a3030-72b2-4223-b95e-c3997a8eb0f6","Type":"ContainerStarted","Data":"59d770ac32b29db330d9d03ff2a4e039a22e00206091186d6c4afbeeefce25ef"}
Oct 31 00:28:09.109756 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.109733    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8p6vl" event={"ID":"b9fad5d0-0d3a-4ebe-a07b-1899400059a7","Type":"ContainerStarted","Data":"efc2ad77731ea8e8e9d413d9c176a4569eacfea00892764a8acba158ce419e5f"}
Oct 31 00:28:09.110978 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.110952    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-29ctl" event={"ID":"693a3c1f-8228-4691-bd99-f05fdf784cd4","Type":"ContainerStarted","Data":"f653f495285a89d3f558729b86f578d22e2e440db158445246c22cc032cf67db"}
Oct 31 00:28:09.227320 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.227257    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:23:08 +0000 UTC" deadline="2028-04-29 07:49:48.005127585 +0000 UTC"
Oct 31 00:28:09.227320 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.227315    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13111h21m38.777817088s"
Oct 31 00:28:09.733738 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.733690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:09.734004 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:09.733935    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:09.734086 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:09.734058    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret podName:261b0474-eb3d-4f3c-b33d-01856306e928 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:11.734037961 +0000 UTC m=+6.070293150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret") pod "global-pull-secret-syncer-wdzhr" (UID: "261b0474-eb3d-4f3c-b33d-01856306e928") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:09.834515 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.834467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:09.834707 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:09.834645    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:09.834772 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:09.834709    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs podName:0e67064c-ba5d-4780-9257-9df20f304f29 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:11.834690113 +0000 UTC m=+6.170945293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs") pod "network-metrics-daemon-7smc9" (UID: "0e67064c-ba5d-4780-9257-9df20f304f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:09.935147 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:09.935082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5k6g\" (UniqueName: \"kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g\") pod \"network-check-target-pdpbv\" (UID: \"d48c64d8-b46d-48e0-830e-d9ec6a92ff06\") " pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:09.935392 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:09.935315    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:09.935392 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:09.935342    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:09.935392 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:09.935356    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l5k6g for pod openshift-network-diagnostics/network-check-target-pdpbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:09.935587 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:09.935420    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g podName:d48c64d8-b46d-48e0-830e-d9ec6a92ff06 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:11.935401406 +0000 UTC m=+6.271656594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5k6g" (UniqueName: "kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g") pod "network-check-target-pdpbv" (UID: "d48c64d8-b46d-48e0-830e-d9ec6a92ff06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:10.089308 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:10.089265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:10.089507 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:10.089273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:10.089507 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:10.089423    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:10.089624 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:10.089532    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:10.089624 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:10.089601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:10.089727 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:10.089675    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:10.227558 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:10.227511    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:23:08 +0000 UTC" deadline="2028-05-01 10:43:09.159947198 +0000 UTC"
Oct 31 00:28:10.227558 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:10.227557    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13162h14m58.932396161s"
Oct 31 00:28:11.750895 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:11.750571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:11.750895 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:11.750766    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:11.750895 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:11.750839    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret podName:261b0474-eb3d-4f3c-b33d-01856306e928 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:15.750820094 +0000 UTC m=+10.087075288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret") pod "global-pull-secret-syncer-wdzhr" (UID: "261b0474-eb3d-4f3c-b33d-01856306e928") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:11.852104 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:11.851448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:11.852104 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:11.851636    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:11.852104 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:11.851721    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs podName:0e67064c-ba5d-4780-9257-9df20f304f29 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:15.851702015 +0000 UTC m=+10.187957205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs") pod "network-metrics-daemon-7smc9" (UID: "0e67064c-ba5d-4780-9257-9df20f304f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:11.951883 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:11.951837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5k6g\" (UniqueName: \"kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g\") pod \"network-check-target-pdpbv\" (UID: \"d48c64d8-b46d-48e0-830e-d9ec6a92ff06\") " pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:11.952108 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:11.952019    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:11.952108 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:11.952043    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:11.952108 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:11.952058    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l5k6g for pod openshift-network-diagnostics/network-check-target-pdpbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:11.952315 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:11.952139    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g podName:d48c64d8-b46d-48e0-830e-d9ec6a92ff06 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:15.952117649 +0000 UTC m=+10.288372846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5k6g" (UniqueName: "kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g") pod "network-check-target-pdpbv" (UID: "d48c64d8-b46d-48e0-830e-d9ec6a92ff06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:12.090784 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:12.090316    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:12.090784 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:12.090497    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:12.090784 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:12.090556    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:12.090784 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:12.090634    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:12.090784 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:12.090673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:12.090784 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:12.090735    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:12.143704 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:12.143585    2577 generic.go:356] "Generic (PLEG): container finished" podID="4919af72af1b023905c8a7373620fc2c" containerID="03a9b1d044b12443a24cd27838bf3462545f1a6cd800867db806a94b2a83b5bb" exitCode=0
Oct 31 00:28:12.143704 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:12.143648    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal" event={"ID":"4919af72af1b023905c8a7373620fc2c","Type":"ContainerDied","Data":"03a9b1d044b12443a24cd27838bf3462545f1a6cd800867db806a94b2a83b5bb"}
Oct 31 00:28:13.150336 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:13.149731    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal" event={"ID":"859c761ccba0302346f6d0c201b0e803","Type":"ContainerStarted","Data":"de4379c3480812a1238e2198b0f2eeb00b8faec8d5a2e95f6cceb1e7297b1f22"}
Oct 31 00:28:13.155463 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:13.155423    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal" event={"ID":"4919af72af1b023905c8a7373620fc2c","Type":"ContainerStarted","Data":"a22cba39c74c7ad19f981d728ea0d46de7f88b00ca1034851d17f8bb7f781c5b"}
Oct 31 00:28:13.159544 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:13.158790    2577 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="a43b1a197a33502d9485f0d793c3ba4d9fc5d2e6a45ae2f7e63a09ef14e5fe2f" exitCode=0
Oct 31 00:28:13.159544 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:13.158841    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8q7j" event={"ID":"b1402f00-5adc-44cb-961c-a2556e51529b","Type":"ContainerDied","Data":"a43b1a197a33502d9485f0d793c3ba4d9fc5d2e6a45ae2f7e63a09ef14e5fe2f"}
Oct 31 00:28:13.281245 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:13.279992    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal" podStartSLOduration=5.279971719 podStartE2EDuration="5.279971719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:13.279612315 +0000 UTC m=+7.615867514" watchObservedRunningTime="2026-10-31 00:28:13.279971719 +0000 UTC m=+7.616226917"
Oct 31 00:28:13.546613 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:13.546553    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal" podStartSLOduration=5.546531116 podStartE2EDuration="5.546531116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:13.408519642 +0000 UTC m=+7.744774840" watchObservedRunningTime="2026-10-31 00:28:13.546531116 +0000 UTC m=+7.882786317"
Oct 31 00:28:14.089784 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:14.089742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:14.089982 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:14.089876    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:14.089982 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:14.089955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:14.090148 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:14.090039    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:14.090148 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:14.090095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:14.090258 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:14.090188    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:14.165510 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:14.165166    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l64l2" event={"ID":"7ab38846-5d17-4bf1-92c2-ac823690fc98","Type":"ContainerStarted","Data":"6d6dc37928ada682b11168fa9fa61a2aef16d87d1cc7b32e55980ea82bec137f"}
Oct 31 00:28:14.297275 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:14.296610    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l64l2" podStartSLOduration=2.391415766 podStartE2EDuration="7.296589558s" totalImagesPullingTime="4.905173792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:14.295907355 +0000 UTC m=+8.632162554" watchObservedRunningTime="2026-10-31 00:28:14.296589558 +0000 UTC m=+8.632844758"
Oct 31 00:28:15.170072 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:15.169956    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fvksv" event={"ID":"f5703958-5751-4c4d-b979-01585c84cd46","Type":"ContainerStarted","Data":"0d127343777f2ed362562d7938b3918394f406a8baa77b2761b2c52120505ecc"}
Oct 31 00:28:15.782395 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:15.782354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:15.782598 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:15.782565    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:15.782665 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:15.782627    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret podName:261b0474-eb3d-4f3c-b33d-01856306e928 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:23.782608861 +0000 UTC m=+18.118864053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret") pod "global-pull-secret-syncer-wdzhr" (UID: "261b0474-eb3d-4f3c-b33d-01856306e928") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:15.883091 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:15.883037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:15.883280 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:15.883217    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:15.883280 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:15.883279    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs podName:0e67064c-ba5d-4780-9257-9df20f304f29 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:23.883260062 +0000 UTC m=+18.219515250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs") pod "network-metrics-daemon-7smc9" (UID: "0e67064c-ba5d-4780-9257-9df20f304f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:15.983772 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:15.983734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5k6g\" (UniqueName: \"kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g\") pod \"network-check-target-pdpbv\" (UID: \"d48c64d8-b46d-48e0-830e-d9ec6a92ff06\") " pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:15.983950 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:15.983906    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:15.983950 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:15.983932    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:15.983950 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:15.983946    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l5k6g for pod openshift-network-diagnostics/network-check-target-pdpbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:15.984105 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:15.984009    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g podName:d48c64d8-b46d-48e0-830e-d9ec6a92ff06 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:23.983988518 +0000 UTC m=+18.320243706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5k6g" (UniqueName: "kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g") pod "network-check-target-pdpbv" (UID: "d48c64d8-b46d-48e0-830e-d9ec6a92ff06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:16.088972 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:16.088889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:16.089170 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:16.089015    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:16.089170 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:16.089123    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:16.089289 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:16.089242    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:16.089351 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:16.089306    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:16.089407 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:16.089387    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:16.177083 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:16.177039    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s" event={"ID":"34d3cbf5-134e-4f86-8b33-04ef38cdb685","Type":"ContainerStarted","Data":"e8772108358d58521dab7830c78cd5e730847444cf3f6206ff360c8a42024f0b"}
Oct 31 00:28:17.182079 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:17.182026    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-76vsc" event={"ID":"d7ef07d0-5483-4176-bfb2-4e03c9be4498","Type":"ContainerStarted","Data":"5138395e49ce5b4e94b3d6aed9035abd3b5a49a6625ea5e54f27bea7e75f171d"}
Oct 31 00:28:17.319070 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:17.318958    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fvksv" podStartSLOduration=4.123083051 podStartE2EDuration="10.318938851s" totalImagesPullingTime="6.1958558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:15.299013622 +0000 UTC m=+9.635268819" watchObservedRunningTime="2026-10-31 00:28:17.318938851 +0000 UTC m=+11.655194046"
Oct 31 00:28:17.319232 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:17.319117    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-76vsc" podStartSLOduration=1.718913389 podStartE2EDuration="10.31911002s" totalImagesPullingTime="8.600196631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:17.318456918 +0000 UTC m=+11.654712111" watchObservedRunningTime="2026-10-31 00:28:17.31911002 +0000 UTC m=+11.655365214"
Oct 31 00:28:17.701050 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:17.701007    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:28:18.042940 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.042726    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:18.044072 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.043901    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:18.089176 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.089150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:18.089316 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.089264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:18.089316 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:18.089276    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:18.089415 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:18.089378    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:18.089453 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.089444    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:18.089532 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:18.089513    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:18.095275 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.095168    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:28:17.701027525Z","UUID":"0ed4c956-189a-4199-bfb9-430732eb3314","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:28:18.097053 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.096866    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:28:18.097053 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.096940    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:28:18.187337 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.187313    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8q7j" event={"ID":"b1402f00-5adc-44cb-961c-a2556e51529b","Type":"ContainerStarted","Data":"5609e590a06a63efcf736e76edfab193b90cf8449d45a89991f604131f6cb01a"}
Oct 31 00:28:18.190155 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.190094    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s" event={"ID":"34d3cbf5-134e-4f86-8b33-04ef38cdb685","Type":"ContainerStarted","Data":"12d43fa27181c3b1eebabf8040e5cccb5d0a82f1db5f5bbbec42c7d0d25f6e89"}
Oct 31 00:28:18.191651 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.191620    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-29ctl" event={"ID":"693a3c1f-8228-4691-bd99-f05fdf784cd4","Type":"ContainerStarted","Data":"9a97bea76fcc600a6ef62c73476ee60ab380ea9dcebd6dac5a269e1975cf2bb5"}
Oct 31 00:28:18.435537 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:18.435447    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-29ctl" podStartSLOduration=2.127241015 podStartE2EDuration="11.435429738s" totalImagesPullingTime="9.308188723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:18.435064264 +0000 UTC m=+12.771319476" watchObservedRunningTime="2026-10-31 00:28:18.435429738 +0000 UTC m=+12.771684930"
Oct 31 00:28:19.197804 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:19.197560    2577 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="5609e590a06a63efcf736e76edfab193b90cf8449d45a89991f604131f6cb01a" exitCode=0
Oct 31 00:28:19.198275 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:19.198202    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8q7j" event={"ID":"b1402f00-5adc-44cb-961c-a2556e51529b","Type":"ContainerDied","Data":"5609e590a06a63efcf736e76edfab193b90cf8449d45a89991f604131f6cb01a"}
Oct 31 00:28:20.087700 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:20.087656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:20.087700 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:20.087704    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:20.087910 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:20.087676    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:20.087910 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:20.087809    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:20.088018 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:20.087928    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:20.088018 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:20.087997    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:20.201535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:20.201488    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sxssn" event={"ID":"c8302492-d696-40ad-bffd-431b35319e22","Type":"ContainerStarted","Data":"6e590df9dd872c7c4805b357e495ae9184711c32424b43916b0e18490991c4c2"}
Oct 31 00:28:20.203807 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:20.203778    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s" event={"ID":"34d3cbf5-134e-4f86-8b33-04ef38cdb685","Type":"ContainerStarted","Data":"6f58d9d60b5d53abf0bae16a0136573d9a6d6bd79b174758c3f32e3c21d8f3d3"}
Oct 31 00:28:20.205228 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:20.205204    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8p6vl" event={"ID":"b9fad5d0-0d3a-4ebe-a07b-1899400059a7","Type":"ContainerStarted","Data":"0a65dfe68306633f29bdcf7f2b26680045edc83e2d61bb4e3cfff585fe752490"}
Oct 31 00:28:20.333966 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:20.333919    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sxssn" podStartSLOduration=2.462209853 podStartE2EDuration="13.333907517s" totalImagesPullingTime="10.871697664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:20.333858145 +0000 UTC m=+14.670113338" watchObservedRunningTime="2026-10-31 00:28:20.333907517 +0000 UTC m=+14.670162710"
Oct 31 00:28:20.452363 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:20.452267    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8p6vl" podStartSLOduration=4.25397613 podStartE2EDuration="13.452250055s" totalImagesPullingTime="9.198273925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:20.451921999 +0000 UTC m=+14.788177194" watchObservedRunningTime="2026-10-31 00:28:20.452250055 +0000 UTC m=+14.788505250"
Oct 31 00:28:20.576359 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:20.576296    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s" podStartSLOduration=3.107216957 podStartE2EDuration="13.57627579s" totalImagesPullingTime="10.469058833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:20.575259748 +0000 UTC m=+14.911514943" watchObservedRunningTime="2026-10-31 00:28:20.57627579 +0000 UTC m=+14.912530985"
Oct 31 00:28:21.088145 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:21.088118    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:21.088299 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:21.088235    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:21.209629 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:21.209595    2577 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="509c859dc7abbaae7d5ed8dc71f1944b1de2064b1bab095f1322199b8be982b7" exitCode=0
Oct 31 00:28:21.210158 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:21.209723    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8q7j" event={"ID":"b1402f00-5adc-44cb-961c-a2556e51529b","Type":"ContainerDied","Data":"509c859dc7abbaae7d5ed8dc71f1944b1de2064b1bab095f1322199b8be982b7"}
Oct 31 00:28:22.087912 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:22.087881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:22.088104 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:22.088060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:22.088104 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:22.088062    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:22.088223 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:22.088179    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:23.088247 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:23.088210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:23.089064 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:23.088340    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:23.217210 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:23.217172    2577 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="a10696b588f51c2de21f024adcd69c0af4f1227ceaae7de50f22cd3538b0a50d" exitCode=0
Oct 31 00:28:23.217385 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:23.217233    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8q7j" event={"ID":"b1402f00-5adc-44cb-961c-a2556e51529b","Type":"ContainerDied","Data":"a10696b588f51c2de21f024adcd69c0af4f1227ceaae7de50f22cd3538b0a50d"}
Oct 31 00:28:23.849099 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:23.849058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:23.849258 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:23.849190    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:23.849258 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:23.849249    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret podName:261b0474-eb3d-4f3c-b33d-01856306e928 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:39.849234606 +0000 UTC m=+34.185489782 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret") pod "global-pull-secret-syncer-wdzhr" (UID: "261b0474-eb3d-4f3c-b33d-01856306e928") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:28:23.950017 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:23.949892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:23.950138 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:23.950059    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:23.950189 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:23.950157    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs podName:0e67064c-ba5d-4780-9257-9df20f304f29 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:39.950136183 +0000 UTC m=+34.286391369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs") pod "network-metrics-daemon-7smc9" (UID: "0e67064c-ba5d-4780-9257-9df20f304f29") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:28:24.050849 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:24.050769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5k6g\" (UniqueName: \"kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g\") pod \"network-check-target-pdpbv\" (UID: \"d48c64d8-b46d-48e0-830e-d9ec6a92ff06\") " pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:24.051026 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:24.050965    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:28:24.051026 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:24.050988    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:28:24.051026 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:24.051004    2577 projected.go:196] Error preparing data for projected volume kube-api-access-l5k6g for pod openshift-network-diagnostics/network-check-target-pdpbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:24.051156 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:24.051081    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g podName:d48c64d8-b46d-48e0-830e-d9ec6a92ff06 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.05106304 +0000 UTC m=+34.387318218 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l5k6g" (UniqueName: "kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g") pod "network-check-target-pdpbv" (UID: "d48c64d8-b46d-48e0-830e-d9ec6a92ff06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:28:24.087940 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:24.087920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:24.088022 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:24.087934    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:24.088088 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:24.088021    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:24.088167 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:24.088149    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:24.221528 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:24.221503    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg" event={"ID":"a27a3030-72b2-4223-b95e-c3997a8eb0f6","Type":"ContainerStarted","Data":"5c7778da1daa9c61865d38530788b38a0112d7282c344e316940a422e3a38770"}
Oct 31 00:28:24.221904 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:24.221584    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg" event={"ID":"a27a3030-72b2-4223-b95e-c3997a8eb0f6","Type":"ContainerStarted","Data":"de46db68d4f6b2967e9a40b6dc2648e05816cb8b8abb1599430a6a27be1c02c6"}
Oct 31 00:28:24.221904 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:24.221637    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg" event={"ID":"a27a3030-72b2-4223-b95e-c3997a8eb0f6","Type":"ContainerStarted","Data":"2ff29fdf17d6b22107ec1c3308225199dce938a5e1fe860f3d702d51cf6ca5ba"}
Oct 31 00:28:25.088068 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:25.087869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:25.088220 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:25.088161    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:25.227190 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:25.227155    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg" event={"ID":"a27a3030-72b2-4223-b95e-c3997a8eb0f6","Type":"ContainerStarted","Data":"f7617d36811c50b0d66a30c189626edf93f1f694a1e4e7e69e7585eecf78634f"}
Oct 31 00:28:25.227625 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:25.227245    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg" event={"ID":"a27a3030-72b2-4223-b95e-c3997a8eb0f6","Type":"ContainerStarted","Data":"5a9fcecf404bc819b34c3ec64447c782f7db18b44d8e570deec2d2fcf6e4f7b7"}
Oct 31 00:28:25.227625 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:25.227291    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg" event={"ID":"a27a3030-72b2-4223-b95e-c3997a8eb0f6","Type":"ContainerStarted","Data":"02143f9da3dd3ce5ac250049ed2ef0f5b6400420466d68ad5159a1f6362ac04b"}
Oct 31 00:28:26.088242 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:26.088213    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:26.088389 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:26.088299    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:26.088389 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:26.088343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:26.088389 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:26.088385    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:26.232115 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:26.232078    2577 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="cf7c7e86295c6d7d80f2c411f27774e2fc3121dac236ccb8a55ed12c640048f7" exitCode=0
Oct 31 00:28:26.232617 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:26.232129    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8q7j" event={"ID":"b1402f00-5adc-44cb-961c-a2556e51529b","Type":"ContainerDied","Data":"cf7c7e86295c6d7d80f2c411f27774e2fc3121dac236ccb8a55ed12c640048f7"}
Oct 31 00:28:27.087277 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:27.087246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:27.087431 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:27.087333    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:27.237956 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:27.237919    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg" event={"ID":"a27a3030-72b2-4223-b95e-c3997a8eb0f6","Type":"ContainerStarted","Data":"7fb6428fac3b08234d0725bdcceacb9d711cc20bfd29d8ff66975a0a6db924d5"}
Oct 31 00:28:27.240754 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:27.240721    2577 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="fa6f1d14d81dc2c1b28b9a8c1845f34ef789d30ee19b96bfcd6d37d82b46270b" exitCode=0
Oct 31 00:28:27.240885 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:27.240762    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8q7j" event={"ID":"b1402f00-5adc-44cb-961c-a2556e51529b","Type":"ContainerDied","Data":"fa6f1d14d81dc2c1b28b9a8c1845f34ef789d30ee19b96bfcd6d37d82b46270b"}
Oct 31 00:28:28.088198 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:28.088168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:28.088331 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:28.088286    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:28.088501 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:28.088340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:28.088501 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:28.088425    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:28.245490 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:28.245460    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h8q7j" event={"ID":"b1402f00-5adc-44cb-961c-a2556e51529b","Type":"ContainerStarted","Data":"c01c56ff23bb2a8eac1457ef47dd45f8212120ee8b3a6a99bbd69b30943d259a"}
Oct 31 00:28:28.423238 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:28.423148    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h8q7j" podStartSLOduration=8.063771809 podStartE2EDuration="21.42313289s" totalImagesPullingTime="13.359361081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:28.422835734 +0000 UTC m=+22.759090927" watchObservedRunningTime="2026-10-31 00:28:28.42313289 +0000 UTC m=+22.759388081"
Oct 31 00:28:29.087915 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:29.087885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:29.088077 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:29.087979    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:29.251203 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:29.251162    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg" event={"ID":"a27a3030-72b2-4223-b95e-c3997a8eb0f6","Type":"ContainerStarted","Data":"c1673227430d01a82ebc84f19661dae6e1f27eeabc43f4aa63f2575226c4ef79"}
Oct 31 00:28:29.460194 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:29.460108    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg" podStartSLOduration=7.027576996 podStartE2EDuration="22.460097004s" totalImagesPullingTime="15.432520008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:29.434493768 +0000 UTC m=+23.770748962" watchObservedRunningTime="2026-10-31 00:28:29.460097004 +0000 UTC m=+23.796352197"
Oct 31 00:28:30.087982 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:30.087955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:30.088151 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:30.088096    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:30.088199 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:30.088152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:30.088293 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:30.088272    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:30.253088 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:30.253058    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:30.253533 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:30.253138    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:30.253533 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:30.253180    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:30.268717 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:30.268580    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:30.268850 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:30.268828    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:31.087310 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:31.087283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:31.087480 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:31.087397    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:31.301192 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:31.301160    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wdzhr"]
Oct 31 00:28:31.301544 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:31.301252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:31.301544 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:31.301339    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:31.304299 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:31.304276    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7smc9"]
Oct 31 00:28:31.304390 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:31.304357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:31.304453 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:31.304438    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:31.304719 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:31.304701    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pdpbv"]
Oct 31 00:28:31.304778 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:31.304769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:31.304847 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:31.304833    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:33.087895 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:33.087869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:33.087895 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:33.087879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:33.088306 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:33.087960    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:33.088306 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:33.087998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:33.088306 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:33.088083    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:33.088306 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:33.088133    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:35.087700 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:35.087676    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:35.088099 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:35.087701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:35.088099 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:35.087782    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7smc9" podUID="0e67064c-ba5d-4780-9257-9df20f304f29"
Oct 31 00:28:35.088099 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:35.087792    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:35.088099 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:35.087847    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pdpbv" podUID="d48c64d8-b46d-48e0-830e-d9ec6a92ff06"
Oct 31 00:28:35.088099 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:35.087919    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdzhr" podUID="261b0474-eb3d-4f3c-b33d-01856306e928"
Oct 31 00:28:36.903084 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:36.902989    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeReady"
Oct 31 00:28:36.903409 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:36.903105    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 00:28:37.053250 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.053212    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hvfpm"]
Oct 31 00:28:37.077407 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.077374    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tkpbr"]
Oct 31 00:28:37.077567 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.077485    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hvfpm"
Oct 31 00:28:37.093470 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.093450    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9lmll"]
Oct 31 00:28:37.093580 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.093496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tkpbr"
Oct 31 00:28:37.115591 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.115537    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cfc76855c-xm22c"]
Oct 31 00:28:37.115707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.115659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:37.115707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.115674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:37.115707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.115679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:37.116101 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.115931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:37.142391 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.142364    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"]
Oct 31 00:28:37.142499 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.142465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.153245 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.151283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z89wv\" (UniqueName: \"kubernetes.io/projected/dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed-kube-api-access-z89wv\") pod \"kas-connection-checker-685895c7c9-hvfpm\" (UID: \"dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed\") " pod="kube-system/kas-connection-checker-685895c7c9-hvfpm"
Oct 31 00:28:37.158746 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.158725    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4sch4"]
Oct 31 00:28:37.158900 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.158848    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.160835 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.160795    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 00:28:37.160835 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.160814    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 00:28:37.179352 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.179276    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-55c864699c-n4cfn"]
Oct 31 00:28:37.179613 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.179593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.181704 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.181594    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:28:37.181908 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.181876    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 00:28:37.181989 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.181905    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-n47jx\""
Oct 31 00:28:37.181989 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.181921    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9b4md\""
Oct 31 00:28:37.182218 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.182199    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6z87r\""
Oct 31 00:28:37.182296 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.182257    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-prxzz\""
Oct 31 00:28:37.182364 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.182351    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:28:37.193585 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.193567    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt"]
Oct 31 00:28:37.193686 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.193663    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.211539 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.211519    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8tt4s"]
Oct 31 00:28:37.211643 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.211628    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt"
Oct 31 00:28:37.234414 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.234389    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp"]
Oct 31 00:28:37.234552 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.234499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8tt4s"
Oct 31 00:28:37.240119 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.240103    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:37.240248 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.240232    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 00:28:37.240321 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.240305    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:28:37.240376 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.240323    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:28:37.240376 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.240340    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:37.240376 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.240308    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 00:28:37.240515 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.240378    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:28:37.240515 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.240408    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:28:37.240515 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.240436    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 00:28:37.246966 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.246938    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"]
Oct 31 00:28:37.247072 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.247009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp"
Oct 31 00:28:37.251720 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.251703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-config\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.251802 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.251728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76cd3516-aff9-4de2-a063-c30407f68564-ca-trust-extracted\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.251802 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.251748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z89wv\" (UniqueName: \"kubernetes.io/projected/dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed-kube-api-access-z89wv\") pod \"kas-connection-checker-685895c7c9-hvfpm\" (UID: \"dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed\") " pod="kube-system/kas-connection-checker-685895c7c9-hvfpm"
Oct 31 00:28:37.251802 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.251767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-image-registry-private-configuration\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.251917 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.251844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.251917 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.251886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44dtj\" (UniqueName: \"kubernetes.io/projected/c44bba77-c0b2-49cc-92c1-836cf8013950-kube-api-access-44dtj\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.251917 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.251910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-tmp\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.252029 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.251934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-registry-certificates\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.252029 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.251975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-service-ca-bundle\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.252116 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-trusted-ca\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.252116 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-bound-sa-token\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.252116 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfmz7\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-kube-api-access-xfmz7\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.252116 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvtlp\" (UniqueName: \"kubernetes.io/projected/f53b8d98-bbb7-4108-a65d-6a15ab782bee-kube-api-access-dvtlp\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.252236 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-installation-pull-secrets\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.252236 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d543da15-0c67-4912-a39e-04235a6afa20-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:37.252297 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:37.252337 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2gk\" (UniqueName: \"kubernetes.io/projected/81be0f35-6496-46d5-a27b-46bd185e3993-kube-api-access-jb2gk\") pod \"kas-connection-checker-685895c7c9-tkpbr\" (UID: \"81be0f35-6496-46d5-a27b-46bd185e3993\") " pod="kube-system/kas-connection-checker-685895c7c9-tkpbr"
Oct 31 00:28:37.252386 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f53b8d98-bbb7-4108-a65d-6a15ab782bee-serving-cert\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.252454 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c44bba77-c0b2-49cc-92c1-836cf8013950-serving-cert\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.252490 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-trusted-ca\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.252542 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.252542 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.252529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-snapshots\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.259295 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.259277    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:37.259394 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.259354    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:37.259394 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.259284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:37.259394 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.259380    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"]
Oct 31 00:28:37.275319 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.275292    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"]
Oct 31 00:28:37.275421 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.275408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:37.281446 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.281426    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"]
Oct 31 00:28:37.281540 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.281521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:37.294103 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.294074    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"]
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296594    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296642    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296665    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296694    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296709    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296710    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-nx2tp\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296754    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296766    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296769    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296804    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296830    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296860    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296872    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bjgrg\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296938    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.296666    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 00:28:37.296995 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.297007    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 00:28:37.297808 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.297205    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pkx6z\""
Oct 31 00:28:37.297808 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.297259    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 00:28:37.297808 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.297271    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 00:28:37.297808 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.297291    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qjlzt\""
Oct 31 00:28:37.301459 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.301444    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-z9m2w\""
Oct 31 00:28:37.310859 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.310842    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-grc8r"]
Oct 31 00:28:37.310972 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.310958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:37.313437 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.313418    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 00:28:37.313531 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.313464    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:37.313582 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.313542    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:37.313618 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.313592    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bz5sd\""
Oct 31 00:28:37.313676 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.313658    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 00:28:37.317225 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.317206    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jmdnc"]
Oct 31 00:28:37.317345 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.317332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.320898 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.320878    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z89wv\" (UniqueName: \"kubernetes.io/projected/dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed-kube-api-access-z89wv\") pod \"kas-connection-checker-685895c7c9-hvfpm\" (UID: \"dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed\") " pod="kube-system/kas-connection-checker-685895c7c9-hvfpm"
Oct 31 00:28:37.323344 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323328    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tkpbr"]
Oct 31 00:28:37.323407 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323365    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hvfpm"]
Oct 31 00:28:37.323407 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323387    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9lmll"]
Oct 31 00:28:37.323407 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323405    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt"]
Oct 31 00:28:37.323513 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323422    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"]
Oct 31 00:28:37.323513 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323436    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4sch4"]
Oct 31 00:28:37.323513 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:37.324946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323448    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55c864699c-n4cfn"]
Oct 31 00:28:37.324946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323782    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cfc76855c-xm22c"]
Oct 31 00:28:37.324946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323809    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"]
Oct 31 00:28:37.324946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323831    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8tt4s"]
Oct 31 00:28:37.324946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323850    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp"]
Oct 31 00:28:37.324946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323869    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"]
Oct 31 00:28:37.324946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323889    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jmdnc"]
Oct 31 00:28:37.324946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323907    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"]
Oct 31 00:28:37.324946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323934    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"]
Oct 31 00:28:37.324946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323952    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-grc8r"]
Oct 31 00:28:37.324946 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.323976    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"]
Oct 31 00:28:37.330604 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.330583    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 00:28:37.330699 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.330587    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 00:28:37.330762 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.330694    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 00:28:37.330945 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.330932    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z6hfz\""
Oct 31 00:28:37.331121 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.331107    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 00:28:37.335692 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.335677    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 00:28:37.335969 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.335957    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 00:28:37.336111 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.336017    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:37.336210 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.336184    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:37.336350 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.336336    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vk6nb\""
Oct 31 00:28:37.353555 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-image-registry-private-configuration\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.353621 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfmz7\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-kube-api-access-xfmz7\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.353621 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76cd3516-aff9-4de2-a063-c30407f68564-ca-trust-extracted\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.353621 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.353756 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvtlp\" (UniqueName: \"kubernetes.io/projected/f53b8d98-bbb7-4108-a65d-6a15ab782bee-kube-api-access-dvtlp\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.353756 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.353683    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:37.353756 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44dtj\" (UniqueName: \"kubernetes.io/projected/c44bba77-c0b2-49cc-92c1-836cf8013950-kube-api-access-44dtj\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.353756 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.353697    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfc76855c-xm22c: secret "image-registry-tls" not found
Oct 31 00:28:37.353756 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-tmp\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.353756 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.353743    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls podName:76cd3516-aff9-4de2-a063-c30407f68564 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:37.853725055 +0000 UTC m=+32.189980241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls") pod "image-registry-6cfc76855c-xm22c" (UID: "76cd3516-aff9-4de2-a063-c30407f68564") : secret "image-registry-tls" not found
Oct 31 00:28:37.354057 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-registry-certificates\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.354057 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-service-ca-bundle\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.354057 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.354057 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:37.354057 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-installation-pull-secrets\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.354057 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.353992    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76cd3516-aff9-4de2-a063-c30407f68564-ca-trust-extracted\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.354057 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354008    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-tmp\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-default-certificate\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.354198    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d5cn\" (UniqueName: \"kubernetes.io/projected/54ecc909-648d-4848-9f49-464285458129-kube-api-access-2d5cn\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.354236    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert podName:d543da15-0c67-4912-a39e-04235a6afa20 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:37.854223591 +0000 UTC m=+32.190478763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9lmll" (UID: "d543da15-0c67-4912-a39e-04235a6afa20") : secret "networking-console-plugin-cert" not found
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdphh\" (UniqueName: \"kubernetes.io/projected/408faab9-57a4-4c18-86ae-68772011de21-kube-api-access-mdphh\") pod \"kas-connection-checker-685895c7c9-8tt4s\" (UID: \"408faab9-57a4-4c18-86ae-68772011de21\") " pod="kube-system/kas-connection-checker-685895c7c9-8tt4s"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c44bba77-c0b2-49cc-92c1-836cf8013950-serving-cert\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/54ecc909-648d-4848-9f49-464285458129-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354375    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-trusted-ca\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9rt\" (UniqueName: \"kubernetes.io/projected/2e209316-e440-4bdf-b2d9-cc79ac3e670f-kube-api-access-tq9rt\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354472    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-registry-certificates\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czwhh\" (UniqueName: \"kubernetes.io/projected/967855d0-8720-446c-a8df-9e2b427533b3-kube-api-access-czwhh\") pod \"network-check-source-6f78d59f99-5dlfp\" (UID: \"967855d0-8720-446c-a8df-9e2b427533b3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp"
Oct 31 00:28:37.354535 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354472    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-service-ca-bundle\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.355262 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t88q\" (UniqueName: \"kubernetes.io/projected/34748730-f975-4008-9a37-dad912c7e930-kube-api-access-7t88q\") pod \"volume-data-source-validator-58c9f5f46d-qjwgt\" (UID: \"34748730-f975-4008-9a37-dad912c7e930\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt"
Oct 31 00:28:37.355262 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.354756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-bound-sa-token\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.355262 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:37.355262 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355073    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/990975fd-2466-4224-acbe-6269e49a9ef9-config\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:37.355262 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-config\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.355262 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n42j6\" (UniqueName: \"kubernetes.io/projected/5266b841-4495-40ff-8277-a8c7b6fdba80-kube-api-access-n42j6\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:37.355531 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/990975fd-2466-4224-acbe-6269e49a9ef9-serving-cert\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:37.355531 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.355531 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-trusted-ca\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.355531 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355465    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.355531 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-stats-auth\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.355531 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355509    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-trusted-ca\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.355531 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d543da15-0c67-4912-a39e-04235a6afa20-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:37.355864 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2gk\" (UniqueName: \"kubernetes.io/projected/81be0f35-6496-46d5-a27b-46bd185e3993-kube-api-access-jb2gk\") pod \"kas-connection-checker-685895c7c9-tkpbr\" (UID: \"81be0f35-6496-46d5-a27b-46bd185e3993\") " pod="kube-system/kas-connection-checker-685895c7c9-tkpbr"
Oct 31 00:28:37.355864 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f53b8d98-bbb7-4108-a65d-6a15ab782bee-serving-cert\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.355864 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmnh\" (UniqueName: \"kubernetes.io/projected/990975fd-2466-4224-acbe-6269e49a9ef9-kube-api-access-cbmnh\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:37.355864 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-snapshots\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.355864 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.355714    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-config\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.356276 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.356254    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-trusted-ca\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.356347 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.356282    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d543da15-0c67-4912-a39e-04235a6afa20-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:37.356565 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.356547    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-image-registry-private-configuration\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.356738 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.356717    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-snapshots\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.356884 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.356867    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-installation-pull-secrets\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.357410 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.357396    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c44bba77-c0b2-49cc-92c1-836cf8013950-serving-cert\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.357971 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.357958    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f53b8d98-bbb7-4108-a65d-6a15ab782bee-serving-cert\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.371297 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.371275    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 00:28:37.371386 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.371301    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:37.387241 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.387224    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:37.387450 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.387436    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:28:37.387518 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.387464    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:28:37.387518 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.387480    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 00:28:37.387518 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.387495    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:28:37.387632 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.387532    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:28:37.387632 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.387600    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:28:37.387632 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.387627    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bpzsc\""
Oct 31 00:28:37.387745 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.387687    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tsjd8\""
Oct 31 00:28:37.387777 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.387755    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5vb9k\""
Oct 31 00:28:37.399958 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.399944    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hvfpm"
Oct 31 00:28:37.422841 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.422814    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44dtj\" (UniqueName: \"kubernetes.io/projected/c44bba77-c0b2-49cc-92c1-836cf8013950-kube-api-access-44dtj\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.422938 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.422867    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-bound-sa-token\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.422938 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.422885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvtlp\" (UniqueName: \"kubernetes.io/projected/f53b8d98-bbb7-4108-a65d-6a15ab782bee-kube-api-access-dvtlp\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.432882 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.422987    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfmz7\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-kube-api-access-xfmz7\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.432882 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.423809    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2gk\" (UniqueName: \"kubernetes.io/projected/81be0f35-6496-46d5-a27b-46bd185e3993-kube-api-access-jb2gk\") pod \"kas-connection-checker-685895c7c9-tkpbr\" (UID: \"81be0f35-6496-46d5-a27b-46bd185e3993\") " pod="kube-system/kas-connection-checker-685895c7c9-tkpbr"
Oct 31 00:28:37.456666 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.456635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rbld\" (UniqueName: \"kubernetes.io/projected/f59717aa-c796-4074-b63f-2fc7e13f3633-kube-api-access-5rbld\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:37.456666 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.456669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.456880 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.456695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:37.456880 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.456737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-bound-sa-token\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.456880 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.456815    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:37.456880 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.456837    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 00:28:37.456880 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.456878    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls podName:54ecc909-648d-4848-9f49-464285458129 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:37.956859244 +0000 UTC m=+32.293114418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r9hjn" (UID: "54ecc909-648d-4848-9f49-464285458129") : secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:37.457144 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.456897    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:37.956887179 +0000 UTC m=+32.293142356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : secret "router-metrics-certs-default" not found
Oct 31 00:28:37.457144 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.456844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-default-certificate\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.457144 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.456931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5cn\" (UniqueName: \"kubernetes.io/projected/54ecc909-648d-4848-9f49-464285458129-kube-api-access-2d5cn\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:37.457144 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.456956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdphh\" (UniqueName: \"kubernetes.io/projected/408faab9-57a4-4c18-86ae-68772011de21-kube-api-access-mdphh\") pod \"kas-connection-checker-685895c7c9-8tt4s\" (UID: \"408faab9-57a4-4c18-86ae-68772011de21\") " pod="kube-system/kas-connection-checker-685895c7c9-8tt4s"
Oct 31 00:28:37.457144 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.456982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/54ecc909-648d-4848-9f49-464285458129-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:37.457144 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:37.457144 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9rt\" (UniqueName: \"kubernetes.io/projected/2e209316-e440-4bdf-b2d9-cc79ac3e670f-kube-api-access-tq9rt\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.457144 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpz4f\" (UniqueName: \"kubernetes.io/projected/db65a944-e395-4d70-9807-c061aecf0584-kube-api-access-rpz4f\") pod \"ingress-canary-jmdnc\" (UID: \"db65a944-e395-4d70-9807-c061aecf0584\") " pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:37.457144 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czwhh\" (UniqueName: \"kubernetes.io/projected/967855d0-8720-446c-a8df-9e2b427533b3-kube-api-access-czwhh\") pod \"network-check-source-6f78d59f99-5dlfp\" (UID: \"967855d0-8720-446c-a8df-9e2b427533b3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp"
Oct 31 00:28:37.457586 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-installation-pull-secrets\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.457586 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ppwn\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-kube-api-access-9ppwn\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.457586 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t88q\" (UniqueName: \"kubernetes.io/projected/34748730-f975-4008-9a37-dad912c7e930-kube-api-access-7t88q\") pod \"volume-data-source-validator-58c9f5f46d-qjwgt\" (UID: \"34748730-f975-4008-9a37-dad912c7e930\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt"
Oct 31 00:28:37.457586 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db65a944-e395-4d70-9807-c061aecf0584-cert\") pod \"ingress-canary-jmdnc\" (UID: \"db65a944-e395-4d70-9807-c061aecf0584\") " pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:37.457586 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a996538-9d06-447f-bf87-e7dbb8f078c6-config-volume\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.457586 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:37.457586 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/990975fd-2466-4224-acbe-6269e49a9ef9-config\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:37.457586 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n42j6\" (UniqueName: \"kubernetes.io/projected/5266b841-4495-40ff-8277-a8c7b6fdba80-kube-api-access-n42j6\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:37.457586 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/990975fd-2466-4224-acbe-6269e49a9ef9-serving-cert\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:37.457586 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.457586 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.457496    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 00:28:37.457997 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.457617    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls podName:5266b841-4495-40ff-8277-a8c7b6fdba80 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:37.95758912 +0000 UTC m=+32.293844293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9h6hk" (UID: "5266b841-4495-40ff-8277-a8c7b6fdba80") : secret "samples-operator-tls" not found
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.458264    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:37.958245611 +0000 UTC m=+32.294500790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : configmap references non-existent config key: service-ca.crt
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457779    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458086    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/990975fd-2466-4224-acbe-6269e49a9ef9-config\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.457553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-stats-auth\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a996538-9d06-447f-bf87-e7dbb8f078c6-tmp-dir\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-trusted-ca\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7lmn\" (UniqueName: \"kubernetes.io/projected/2a996538-9d06-447f-bf87-e7dbb8f078c6-kube-api-access-d7lmn\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-image-registry-private-configuration\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-certificates\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f59717aa-c796-4074-b63f-2fc7e13f3633-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458504    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/54ecc909-648d-4848-9f49-464285458129-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmnh\" (UniqueName: \"kubernetes.io/projected/990975fd-2466-4224-acbe-6269e49a9ef9-kube-api-access-cbmnh\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6511745-7315-463c-bc25-0e3e1fd6bf42-ca-trust-extracted\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.458791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.459679 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.458755    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f59717aa-c796-4074-b63f-2fc7e13f3633-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:37.459782 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.459763    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/990975fd-2466-4224-acbe-6269e49a9ef9-serving-cert\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:37.460019 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.460002    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-default-certificate\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.460217 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.460198    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-stats-auth\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.489085 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.489029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:37.525464 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.525429    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmnh\" (UniqueName: \"kubernetes.io/projected/990975fd-2466-4224-acbe-6269e49a9ef9-kube-api-access-cbmnh\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:37.525771 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.525750    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czwhh\" (UniqueName: \"kubernetes.io/projected/967855d0-8720-446c-a8df-9e2b427533b3-kube-api-access-czwhh\") pod \"network-check-source-6f78d59f99-5dlfp\" (UID: \"967855d0-8720-446c-a8df-9e2b427533b3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp"
Oct 31 00:28:37.526116 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.526090    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t88q\" (UniqueName: \"kubernetes.io/projected/34748730-f975-4008-9a37-dad912c7e930-kube-api-access-7t88q\") pod \"volume-data-source-validator-58c9f5f46d-qjwgt\" (UID: \"34748730-f975-4008-9a37-dad912c7e930\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt"
Oct 31 00:28:37.526200 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.526174    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5cn\" (UniqueName: \"kubernetes.io/projected/54ecc909-648d-4848-9f49-464285458129-kube-api-access-2d5cn\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:37.526630 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.526607    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n42j6\" (UniqueName: \"kubernetes.io/projected/5266b841-4495-40ff-8277-a8c7b6fdba80-kube-api-access-n42j6\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:37.526852 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.526837    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdphh\" (UniqueName: \"kubernetes.io/projected/408faab9-57a4-4c18-86ae-68772011de21-kube-api-access-mdphh\") pod \"kas-connection-checker-685895c7c9-8tt4s\" (UID: \"408faab9-57a4-4c18-86ae-68772011de21\") " pod="kube-system/kas-connection-checker-685895c7c9-8tt4s"
Oct 31 00:28:37.527120 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.527103    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9rt\" (UniqueName: \"kubernetes.io/projected/2e209316-e440-4bdf-b2d9-cc79ac3e670f-kube-api-access-tq9rt\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.543283 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.543162    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:37.560146 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-certificates\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.560251 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f59717aa-c796-4074-b63f-2fc7e13f3633-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:37.560251 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.560251 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6511745-7315-463c-bc25-0e3e1fd6bf42-ca-trust-extracted\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.560251 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.560443 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f59717aa-c796-4074-b63f-2fc7e13f3633-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:37.560443 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rbld\" (UniqueName: \"kubernetes.io/projected/f59717aa-c796-4074-b63f-2fc7e13f3633-kube-api-access-5rbld\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:37.560443 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-bound-sa-token\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.560443 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpz4f\" (UniqueName: \"kubernetes.io/projected/db65a944-e395-4d70-9807-c061aecf0584-kube-api-access-rpz4f\") pod \"ingress-canary-jmdnc\" (UID: \"db65a944-e395-4d70-9807-c061aecf0584\") " pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:37.560443 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-installation-pull-secrets\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.560443 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ppwn\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-kube-api-access-9ppwn\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.560719 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db65a944-e395-4d70-9807-c061aecf0584-cert\") pod \"ingress-canary-jmdnc\" (UID: \"db65a944-e395-4d70-9807-c061aecf0584\") " pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:37.560719 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a996538-9d06-447f-bf87-e7dbb8f078c6-config-volume\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.560719 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a996538-9d06-447f-bf87-e7dbb8f078c6-tmp-dir\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.560719 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-trusted-ca\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.560719 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7lmn\" (UniqueName: \"kubernetes.io/projected/2a996538-9d06-447f-bf87-e7dbb8f078c6-kube-api-access-d7lmn\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.560719 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.560555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-image-registry-private-configuration\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.561646 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.561621    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-certificates\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.561707 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.561673    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6511745-7315-463c-bc25-0e3e1fd6bf42-ca-trust-extracted\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.561765 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.561727    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:37.561812 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.561779    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls podName:2a996538-9d06-447f-bf87-e7dbb8f078c6 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:38.061762633 +0000 UTC m=+32.398017819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls") pod "dns-default-grc8r" (UID: "2a996538-9d06-447f-bf87-e7dbb8f078c6") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:37.562470 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.562433    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f59717aa-c796-4074-b63f-2fc7e13f3633-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:37.562769 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.562743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a996538-9d06-447f-bf87-e7dbb8f078c6-tmp-dir\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.563025 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.563000    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a996538-9d06-447f-bf87-e7dbb8f078c6-config-volume\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.563135 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.563063    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:37.563135 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.563080    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f66ccc5cb-h7cvd: secret "image-registry-tls" not found
Oct 31 00:28:37.563135 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.563122    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls podName:d6511745-7315-463c-bc25-0e3e1fd6bf42 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:38.063108203 +0000 UTC m=+32.399363395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls") pod "image-registry-7f66ccc5cb-h7cvd" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42") : secret "image-registry-tls" not found
Oct 31 00:28:37.564500 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.564475    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-trusted-ca\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.564725 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.564707    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db65a944-e395-4d70-9807-c061aecf0584-cert\") pod \"ingress-canary-jmdnc\" (UID: \"db65a944-e395-4d70-9807-c061aecf0584\") " pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:37.564796 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.564772    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-installation-pull-secrets\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.565192 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.565169    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-image-registry-private-configuration\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.566004 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.565987    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f59717aa-c796-4074-b63f-2fc7e13f3633-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:37.580139 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.580115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt"
Oct 31 00:28:37.584081 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.584025    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8tt4s"
Oct 31 00:28:37.589080 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.589011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp"
Oct 31 00:28:37.593168 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.593148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:37.634500 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.631931    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-bound-sa-token\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.636630 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.636589    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpz4f\" (UniqueName: \"kubernetes.io/projected/db65a944-e395-4d70-9807-c061aecf0584-kube-api-access-rpz4f\") pod \"ingress-canary-jmdnc\" (UID: \"db65a944-e395-4d70-9807-c061aecf0584\") " pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:37.637029 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.636996    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rbld\" (UniqueName: \"kubernetes.io/projected/f59717aa-c796-4074-b63f-2fc7e13f3633-kube-api-access-5rbld\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:37.637742 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.637682    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7lmn\" (UniqueName: \"kubernetes.io/projected/2a996538-9d06-447f-bf87-e7dbb8f078c6-kube-api-access-d7lmn\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:37.639085 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.639019    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ppwn\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-kube-api-access-9ppwn\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:37.643877 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.643619    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:37.715778 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.715747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tkpbr"
Oct 31 00:28:37.816251 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.816219    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hvfpm"]
Oct 31 00:28:37.819096 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:37.819074    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc22e5f2_92a8_424b_a3a0_c2f99a43b9ed.slice/crio-ff9b9b39519a0df5ce768b87cff7916719e9a31d4bbe44186736fa34333013c0 WatchSource:0}: Error finding container ff9b9b39519a0df5ce768b87cff7916719e9a31d4bbe44186736fa34333013c0: Status 404 returned error can't find the container with id ff9b9b39519a0df5ce768b87cff7916719e9a31d4bbe44186736fa34333013c0
Oct 31 00:28:37.864023 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.863996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:37.864198 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.864075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:37.864198 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.864184    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:37.864309 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.864195    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 00:28:37.864309 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.864206    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfc76855c-xm22c: secret "image-registry-tls" not found
Oct 31 00:28:37.864309 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.864266    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls podName:76cd3516-aff9-4de2-a063-c30407f68564 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:38.864253009 +0000 UTC m=+33.200508182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls") pod "image-registry-6cfc76855c-xm22c" (UID: "76cd3516-aff9-4de2-a063-c30407f68564") : secret "image-registry-tls" not found
Oct 31 00:28:37.864309 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.864283    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert podName:d543da15-0c67-4912-a39e-04235a6afa20 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:38.864274461 +0000 UTC m=+33.200529633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9lmll" (UID: "d543da15-0c67-4912-a39e-04235a6afa20") : secret "networking-console-plugin-cert" not found
Oct 31 00:28:37.872555 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.872526    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"]
Oct 31 00:28:37.875664 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:37.875607    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44bba77_c0b2_49cc_92c1_836cf8013950.slice/crio-92169b77fc7554447d2dd9fb89c3b6c9fb184a3190082a40f1f251f1d1e0588f WatchSource:0}: Error finding container 92169b77fc7554447d2dd9fb89c3b6c9fb184a3190082a40f1f251f1d1e0588f: Status 404 returned error can't find the container with id 92169b77fc7554447d2dd9fb89c3b6c9fb184a3190082a40f1f251f1d1e0588f
Oct 31 00:28:37.924225 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.924199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:37.927461 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.927440    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4sch4"]
Oct 31 00:28:37.929674 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:37.929645    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53b8d98_bbb7_4108_a65d_6a15ab782bee.slice/crio-99cd239f879c4d8945493c1c5fd1bbbc9d46e942c27f545275e820265d06f214 WatchSource:0}: Error finding container 99cd239f879c4d8945493c1c5fd1bbbc9d46e942c27f545275e820265d06f214: Status 404 returned error can't find the container with id 99cd239f879c4d8945493c1c5fd1bbbc9d46e942c27f545275e820265d06f214
Oct 31 00:28:37.964894 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.964855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:37.965019 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.964970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:37.965019 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.964996    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:37.965019 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.965009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.965202 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.965075    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls podName:54ecc909-648d-4848-9f49-464285458129 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:38.965058237 +0000 UTC m=+33.301313422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r9hjn" (UID: "54ecc909-648d-4848-9f49-464285458129") : secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:37.965202 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.965128    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 00:28:37.965202 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.965133    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:38.965121757 +0000 UTC m=+33.301376928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : configmap references non-existent config key: service-ca.crt
Oct 31 00:28:37.965202 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.965199    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls podName:5266b841-4495-40ff-8277-a8c7b6fdba80 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:38.965185623 +0000 UTC m=+33.301440795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9h6hk" (UID: "5266b841-4495-40ff-8277-a8c7b6fdba80") : secret "samples-operator-tls" not found
Oct 31 00:28:37.965397 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.965238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:37.965397 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.965320    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 00:28:37.965397 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:37.965356    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:38.965345008 +0000 UTC m=+33.301600196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : secret "router-metrics-certs-default" not found
Oct 31 00:28:37.974765 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.974732    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt"]
Oct 31 00:28:37.975539 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:37.975510    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8tt4s"]
Oct 31 00:28:37.978794 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:37.978771    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408faab9_57a4_4c18_86ae_68772011de21.slice/crio-2236639900fd66a76022f581d57a34b0c3a890430f3ef984b48b96a5cddd6896 WatchSource:0}: Error finding container 2236639900fd66a76022f581d57a34b0c3a890430f3ef984b48b96a5cddd6896: Status 404 returned error can't find the container with id 2236639900fd66a76022f581d57a34b0c3a890430f3ef984b48b96a5cddd6896
Oct 31 00:28:37.984922 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:37.984896    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34748730_f975_4008_9a37_dad912c7e930.slice/crio-ad0ab66cdd8de4cb2648cfa7d40b02409fb1fd42d5fba5885a60caadaa54d068 WatchSource:0}: Error finding container ad0ab66cdd8de4cb2648cfa7d40b02409fb1fd42d5fba5885a60caadaa54d068: Status 404 returned error can't find the container with id ad0ab66cdd8de4cb2648cfa7d40b02409fb1fd42d5fba5885a60caadaa54d068
Oct 31 00:28:38.006791 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.006051    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"]
Oct 31 00:28:38.009281 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.008895    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp"]
Oct 31 00:28:38.015824 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:38.015153    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967855d0_8720_446c_a8df_9e2b427533b3.slice/crio-4f91ec2ee852545672bc9664a804845aaaac3a56dd03e60e633125fd790c0099 WatchSource:0}: Error finding container 4f91ec2ee852545672bc9664a804845aaaac3a56dd03e60e633125fd790c0099: Status 404 returned error can't find the container with id 4f91ec2ee852545672bc9664a804845aaaac3a56dd03e60e633125fd790c0099
Oct 31 00:28:38.021775 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.021727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:38.050367 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.050331    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jmdnc"]
Oct 31 00:28:38.054589 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:38.054557    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb65a944_e395_4d70_9807_c061aecf0584.slice/crio-a7aea42d2a498166e52d91964f650a8c131a6483cf0a77cd63764dc7f9651079 WatchSource:0}: Error finding container a7aea42d2a498166e52d91964f650a8c131a6483cf0a77cd63764dc7f9651079: Status 404 returned error can't find the container with id a7aea42d2a498166e52d91964f650a8c131a6483cf0a77cd63764dc7f9651079
Oct 31 00:28:38.065991 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.065967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:38.066167 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.066148    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:38.066229 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.066162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:38.066229 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.066205    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls podName:2a996538-9d06-447f-bf87-e7dbb8f078c6 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:39.06619151 +0000 UTC m=+33.402446713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls") pod "dns-default-grc8r" (UID: "2a996538-9d06-447f-bf87-e7dbb8f078c6") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:38.066339 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.066291    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:38.066339 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.066305    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f66ccc5cb-h7cvd: secret "image-registry-tls" not found
Oct 31 00:28:38.066417 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.066357    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls podName:d6511745-7315-463c-bc25-0e3e1fd6bf42 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:39.06634183 +0000 UTC m=+33.402597018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls") pod "image-registry-7f66ccc5cb-h7cvd" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42") : secret "image-registry-tls" not found
Oct 31 00:28:38.112442 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.112413    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tkpbr"]
Oct 31 00:28:38.115131 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:38.115106    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81be0f35_6496_46d5_a27b_46bd185e3993.slice/crio-4479da8f52de7c7e460cf7dd7e12df5a099ceb3a815f3d483c29d4c3781924d5 WatchSource:0}: Error finding container 4479da8f52de7c7e460cf7dd7e12df5a099ceb3a815f3d483c29d4c3781924d5: Status 404 returned error can't find the container with id 4479da8f52de7c7e460cf7dd7e12df5a099ceb3a815f3d483c29d4c3781924d5
Oct 31 00:28:38.269576 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.269541    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt" event={"ID":"34748730-f975-4008-9a37-dad912c7e930","Type":"ContainerStarted","Data":"ad0ab66cdd8de4cb2648cfa7d40b02409fb1fd42d5fba5885a60caadaa54d068"}
Oct 31 00:28:38.270980 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.270950    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hvfpm" event={"ID":"dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed","Type":"ContainerStarted","Data":"80cdda7c7014ca7d10a2164a9bd570670fd468592fa37636063ad62dc70a6e29"}
Oct 31 00:28:38.271130 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.271010    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hvfpm" event={"ID":"dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed","Type":"ContainerStarted","Data":"ff9b9b39519a0df5ce768b87cff7916719e9a31d4bbe44186736fa34333013c0"}
Oct 31 00:28:38.272607 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.272580    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tkpbr" event={"ID":"81be0f35-6496-46d5-a27b-46bd185e3993","Type":"ContainerStarted","Data":"59ec45fa8af0561f19a6fa01d16a9b7b20474cef43841b5b008119819c2cd76f"}
Oct 31 00:28:38.272714 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.272623    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tkpbr" event={"ID":"81be0f35-6496-46d5-a27b-46bd185e3993","Type":"ContainerStarted","Data":"4479da8f52de7c7e460cf7dd7e12df5a099ceb3a815f3d483c29d4c3781924d5"}
Oct 31 00:28:38.273636 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.273615    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jmdnc" event={"ID":"db65a944-e395-4d70-9807-c061aecf0584","Type":"ContainerStarted","Data":"a7aea42d2a498166e52d91964f650a8c131a6483cf0a77cd63764dc7f9651079"}
Oct 31 00:28:38.274688 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.274670    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8" event={"ID":"990975fd-2466-4224-acbe-6269e49a9ef9","Type":"ContainerStarted","Data":"5ce410fcc02e43610380bb82c65b50f64af5c4d76a4d54c987c22b45f2df187e"}
Oct 31 00:28:38.275650 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.275632    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp" event={"ID":"967855d0-8720-446c-a8df-9e2b427533b3","Type":"ContainerStarted","Data":"4f91ec2ee852545672bc9664a804845aaaac3a56dd03e60e633125fd790c0099"}
Oct 31 00:28:38.276679 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.276661    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" event={"ID":"c44bba77-c0b2-49cc-92c1-836cf8013950","Type":"ContainerStarted","Data":"92169b77fc7554447d2dd9fb89c3b6c9fb184a3190082a40f1f251f1d1e0588f"}
Oct 31 00:28:38.277684 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.277666    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4sch4" event={"ID":"f53b8d98-bbb7-4108-a65d-6a15ab782bee","Type":"ContainerStarted","Data":"99cd239f879c4d8945493c1c5fd1bbbc9d46e942c27f545275e820265d06f214"}
Oct 31 00:28:38.278835 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.278816    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8tt4s" event={"ID":"408faab9-57a4-4c18-86ae-68772011de21","Type":"ContainerStarted","Data":"9ca7aaad64fe924e062ac21499272dc35dd1a82583dfe962bf03b0878be0aaf5"}
Oct 31 00:28:38.278909 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.278849    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8tt4s" event={"ID":"408faab9-57a4-4c18-86ae-68772011de21","Type":"ContainerStarted","Data":"2236639900fd66a76022f581d57a34b0c3a890430f3ef984b48b96a5cddd6896"}
Oct 31 00:28:38.321294 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.321269    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"]
Oct 31 00:28:38.324212 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:38.324188    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59717aa_c796_4074_b63f_2fc7e13f3633.slice/crio-5037dff5fe581117ec599ee6a185baa4519a23dad1d40dfda65ed4c47a78c1a4 WatchSource:0}: Error finding container 5037dff5fe581117ec599ee6a185baa4519a23dad1d40dfda65ed4c47a78c1a4: Status 404 returned error can't find the container with id 5037dff5fe581117ec599ee6a185baa4519a23dad1d40dfda65ed4c47a78c1a4
Oct 31 00:28:38.874409 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.874365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:38.874589 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.874423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:38.874589 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.874548    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:38.874589 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.874560    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 00:28:38.874589 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.874571    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfc76855c-xm22c: secret "image-registry-tls" not found
Oct 31 00:28:38.874705 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.874636    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls podName:76cd3516-aff9-4de2-a063-c30407f68564 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.874616804 +0000 UTC m=+35.210871989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls") pod "image-registry-6cfc76855c-xm22c" (UID: "76cd3516-aff9-4de2-a063-c30407f68564") : secret "image-registry-tls" not found
Oct 31 00:28:38.874705 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.874653    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert podName:d543da15-0c67-4912-a39e-04235a6afa20 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.874646979 +0000 UTC m=+35.210902151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9lmll" (UID: "d543da15-0c67-4912-a39e-04235a6afa20") : secret "networking-console-plugin-cert" not found
Oct 31 00:28:38.975112 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.975076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:38.975538 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.975148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:38.975538 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.975240    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 00:28:38.975538 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.975276    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:38.975538 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.975246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:38.975538 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.975323    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.975301718 +0000 UTC m=+35.311556890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : secret "router-metrics-certs-default" not found
Oct 31 00:28:38.975538 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.975335    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 00:28:38.975538 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.975347    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls podName:54ecc909-648d-4848-9f49-464285458129 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.975337322 +0000 UTC m=+35.311592499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r9hjn" (UID: "54ecc909-648d-4848-9f49-464285458129") : secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:38.975538 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.975369    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls podName:5266b841-4495-40ff-8277-a8c7b6fdba80 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.975358494 +0000 UTC m=+35.311613666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9h6hk" (UID: "5266b841-4495-40ff-8277-a8c7b6fdba80") : secret "samples-operator-tls" not found
Oct 31 00:28:38.975538 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:38.975400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:38.975538 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:38.975509    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.975502077 +0000 UTC m=+35.311757250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : configmap references non-existent config key: service-ca.crt
Oct 31 00:28:39.076239 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:39.076196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:39.076445 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:39.076259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:39.076445 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:39.076332    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:39.076445 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:39.076352    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f66ccc5cb-h7cvd: secret "image-registry-tls" not found
Oct 31 00:28:39.076445 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:39.076409    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls podName:d6511745-7315-463c-bc25-0e3e1fd6bf42 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.076393584 +0000 UTC m=+35.412648756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls") pod "image-registry-7f66ccc5cb-h7cvd" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42") : secret "image-registry-tls" not found
Oct 31 00:28:39.076445 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:39.076434    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:39.076708 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:39.076481    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls podName:2a996538-9d06-447f-bf87-e7dbb8f078c6 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.076466714 +0000 UTC m=+35.412721906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls") pod "dns-default-grc8r" (UID: "2a996538-9d06-447f-bf87-e7dbb8f078c6") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:39.289152 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:39.289114    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng" event={"ID":"f59717aa-c796-4074-b63f-2fc7e13f3633","Type":"ContainerStarted","Data":"5037dff5fe581117ec599ee6a185baa4519a23dad1d40dfda65ed4c47a78c1a4"}
Oct 31 00:28:39.439180 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:39.439099    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8tt4s" podStartSLOduration=319.439080138 podStartE2EDuration="5m19.439080138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:39.437371204 +0000 UTC m=+33.773626399" watchObservedRunningTime="2026-10-31 00:28:39.439080138 +0000 UTC m=+33.775335336"
Oct 31 00:28:39.485653 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:39.485615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:39.568785 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:39.568681    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hvfpm" podStartSLOduration=319.568664573 podStartE2EDuration="5m19.568664573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:39.56766362 +0000 UTC m=+33.903918819" watchObservedRunningTime="2026-10-31 00:28:39.568664573 +0000 UTC m=+33.904919767"
Oct 31 00:28:39.884430 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:39.884349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:39.888067 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:39.887540    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:39.900447 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:39.900412    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:39.985774 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:39.985340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:39.985774 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:39.985508    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:28:39.985774 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:39.985570    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs podName:0e67064c-ba5d-4780-9257-9df20f304f29 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:11.985553054 +0000 UTC m=+66.321808231 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs") pod "network-metrics-daemon-7smc9" (UID: "0e67064c-ba5d-4780-9257-9df20f304f29") : secret "metrics-daemon-secret" not found
Oct 31 00:28:40.087504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.087460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5k6g\" (UniqueName: \"kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g\") pod \"network-check-target-pdpbv\" (UID: \"d48c64d8-b46d-48e0-830e-d9ec6a92ff06\") " pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:40.091952 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.091905    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5k6g\" (UniqueName: \"kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g\") pod \"network-check-target-pdpbv\" (UID: \"d48c64d8-b46d-48e0-830e-d9ec6a92ff06\") " pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:40.227601 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.227515    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:40.303781 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.302510    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tkpbr" podStartSLOduration=320.302490803 podStartE2EDuration="5m20.302490803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:39.688194278 +0000 UTC m=+34.024449472" watchObservedRunningTime="2026-10-31 00:28:40.302490803 +0000 UTC m=+34.638745998"
Oct 31 00:28:40.303781 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.303708    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wdzhr"]
Oct 31 00:28:40.317399 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:40.317148    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261b0474_eb3d_4f3c_b33d_01856306e928.slice/crio-a12afa6f7ebfd8eeaba6e68fc9759bfb06eb34678d2cd2af4a1a9c44a43fe539 WatchSource:0}: Error finding container a12afa6f7ebfd8eeaba6e68fc9759bfb06eb34678d2cd2af4a1a9c44a43fe539: Status 404 returned error can't find the container with id a12afa6f7ebfd8eeaba6e68fc9759bfb06eb34678d2cd2af4a1a9c44a43fe539
Oct 31 00:28:40.685922 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.685895    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pdpbv"]
Oct 31 00:28:40.690252 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:40.690219    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48c64d8_b46d_48e0_830e_d9ec6a92ff06.slice/crio-24f3f5a6d858460eadc4193742eacb162f14f47f097c451919ae080b66f127bd WatchSource:0}: Error finding container 24f3f5a6d858460eadc4193742eacb162f14f47f097c451919ae080b66f127bd: Status 404 returned error can't find the container with id 24f3f5a6d858460eadc4193742eacb162f14f47f097c451919ae080b66f127bd
Oct 31 00:28:40.898622 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.898545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:40.898762 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.898629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:40.898762 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.898748    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 00:28:40.898873 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.898761    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:40.898873 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.898807    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfc76855c-xm22c: secret "image-registry-tls" not found
Oct 31 00:28:40.898873 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.898808    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert podName:d543da15-0c67-4912-a39e-04235a6afa20 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:44.898790586 +0000 UTC m=+39.235045772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9lmll" (UID: "d543da15-0c67-4912-a39e-04235a6afa20") : secret "networking-console-plugin-cert" not found
Oct 31 00:28:40.898873 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.898871    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls podName:76cd3516-aff9-4de2-a063-c30407f68564 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:44.898854237 +0000 UTC m=+39.235109415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls") pod "image-registry-6cfc76855c-xm22c" (UID: "76cd3516-aff9-4de2-a063-c30407f68564") : secret "image-registry-tls" not found
Oct 31 00:28:40.999235 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.999196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:40.999713 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.999246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:40.999713 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.999324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:40.999713 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:40.999362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:40.999713 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.999407    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 00:28:40.999713 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.999484    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:44.999465302 +0000 UTC m=+39.335720491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : secret "router-metrics-certs-default" not found
Oct 31 00:28:40.999713 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.999522    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:44.999504889 +0000 UTC m=+39.335760074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : configmap references non-existent config key: service-ca.crt
Oct 31 00:28:40.999713 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.999556    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:40.999713 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.999593    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls podName:54ecc909-648d-4848-9f49-464285458129 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:44.999581222 +0000 UTC m=+39.335836398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r9hjn" (UID: "54ecc909-648d-4848-9f49-464285458129") : secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:40.999713 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.999659    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 00:28:41.000007 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:40.999754    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls podName:5266b841-4495-40ff-8277-a8c7b6fdba80 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:44.999719891 +0000 UTC m=+39.335975064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9h6hk" (UID: "5266b841-4495-40ff-8277-a8c7b6fdba80") : secret "samples-operator-tls" not found
Oct 31 00:28:41.100137 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:41.100104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:41.100292 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:41.100167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:41.100473 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:41.100458    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:41.100520 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:41.100476    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f66ccc5cb-h7cvd: secret "image-registry-tls" not found
Oct 31 00:28:41.100577 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:41.100520    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls podName:d6511745-7315-463c-bc25-0e3e1fd6bf42 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:45.100507015 +0000 UTC m=+39.436762188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls") pod "image-registry-7f66ccc5cb-h7cvd" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42") : secret "image-registry-tls" not found
Oct 31 00:28:41.100577 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:41.100567    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:41.100672 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:41.100587    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls podName:2a996538-9d06-447f-bf87-e7dbb8f078c6 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:45.100580989 +0000 UTC m=+39.436836161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls") pod "dns-default-grc8r" (UID: "2a996538-9d06-447f-bf87-e7dbb8f078c6") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:41.307094 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:41.307020    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt" event={"ID":"34748730-f975-4008-9a37-dad912c7e930","Type":"ContainerStarted","Data":"46bad1c61ade9c1f38a55cce203b3a05d122902831b091a752c51df1131d0636"}
Oct 31 00:28:41.309152 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:41.309121    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pdpbv" event={"ID":"d48c64d8-b46d-48e0-830e-d9ec6a92ff06","Type":"ContainerStarted","Data":"24f3f5a6d858460eadc4193742eacb162f14f47f097c451919ae080b66f127bd"}
Oct 31 00:28:41.311772 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:41.311695    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wdzhr" event={"ID":"261b0474-eb3d-4f3c-b33d-01856306e928","Type":"ContainerStarted","Data":"a12afa6f7ebfd8eeaba6e68fc9759bfb06eb34678d2cd2af4a1a9c44a43fe539"}
Oct 31 00:28:41.428267 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:41.426920    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt" podStartSLOduration=292.381318663 podStartE2EDuration="4m55.426906126s" totalImagesPullingTime="3.045587463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:41.426591134 +0000 UTC m=+35.762846327" watchObservedRunningTime="2026-10-31 00:28:41.426906126 +0000 UTC m=+35.763161319"
Oct 31 00:28:41.607288 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:41.607146    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:41.843848 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:41.843787    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:42.140073 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:42.139973    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:42.332809 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:42.332725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:43.509006 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:43.508971    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:43.762713 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:43.762582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:44.340258 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.339277    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jmdnc" event={"ID":"db65a944-e395-4d70-9807-c061aecf0584","Type":"ContainerStarted","Data":"7f35833ef1f20188b525cc7558417166b8f1e51f94782aafdaf1acb6e4c4cd19"}
Oct 31 00:28:44.347195 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.347117    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8" event={"ID":"990975fd-2466-4224-acbe-6269e49a9ef9","Type":"ContainerStarted","Data":"adcd0085e6dc85733c698f35b6382f36d370640819e168a7e67235cad3370832"}
Oct 31 00:28:44.350096 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.349508    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/0.log"
Oct 31 00:28:44.350096 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.349535    2577 generic.go:356] "Generic (PLEG): container finished" podID="c44bba77-c0b2-49cc-92c1-836cf8013950" containerID="8cd13f8b0ba42b53f2ef0b451a24b706756e3fbbd95c886842779d7291bf956c" exitCode=255
Oct 31 00:28:44.350096 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.349717    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" event={"ID":"c44bba77-c0b2-49cc-92c1-836cf8013950","Type":"ContainerDied","Data":"8cd13f8b0ba42b53f2ef0b451a24b706756e3fbbd95c886842779d7291bf956c"}
Oct 31 00:28:44.350096 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.349813    2577 scope.go:122] "RemoveContainer" containerID="8cd13f8b0ba42b53f2ef0b451a24b706756e3fbbd95c886842779d7291bf956c"
Oct 31 00:28:44.352981 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.352631    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng" event={"ID":"f59717aa-c796-4074-b63f-2fc7e13f3633","Type":"ContainerStarted","Data":"f46f42218a8fac61fc4001c0fec2d5ebd23dff78b8768abea378ffff83a072d1"}
Oct 31 00:28:44.469401 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.469316    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jmdnc" podStartSLOduration=2.545117878 podStartE2EDuration="8.469296426s" totalImagesPullingTime="5.924178548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:44.467379609 +0000 UTC m=+38.803634804" watchObservedRunningTime="2026-10-31 00:28:44.469296426 +0000 UTC m=+38.805551622"
Oct 31 00:28:44.835437 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.835376    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8" podStartSLOduration=293.408646116 podStartE2EDuration="4m58.835354047s" totalImagesPullingTime="5.426707931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:44.714622539 +0000 UTC m=+39.050877733" watchObservedRunningTime="2026-10-31 00:28:44.835354047 +0000 UTC m=+39.171609426"
Oct 31 00:28:44.836500 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.836451    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng" podStartSLOduration=293.189137476 podStartE2EDuration="4m58.836441043s" totalImagesPullingTime="5.647303567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:44.834915167 +0000 UTC m=+39.171170364" watchObservedRunningTime="2026-10-31 00:28:44.836441043 +0000 UTC m=+39.172696242"
Oct 31 00:28:44.951702 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.951652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:44.951895 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:44.951820    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 00:28:44.951895 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:44.951838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:44.951895 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:44.951889    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert podName:d543da15-0c67-4912-a39e-04235a6afa20 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:52.951874771 +0000 UTC m=+47.288129947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9lmll" (UID: "d543da15-0c67-4912-a39e-04235a6afa20") : secret "networking-console-plugin-cert" not found
Oct 31 00:28:44.952074 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:44.951963    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:44.952074 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:44.951987    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfc76855c-xm22c: secret "image-registry-tls" not found
Oct 31 00:28:44.952074 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:44.952056    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls podName:76cd3516-aff9-4de2-a063-c30407f68564 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:52.952025543 +0000 UTC m=+47.288280715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls") pod "image-registry-6cfc76855c-xm22c" (UID: "76cd3516-aff9-4de2-a063-c30407f68564") : secret "image-registry-tls" not found
Oct 31 00:28:45.052962 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.052930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:45.053112 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.052983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:45.053112 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.053085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:45.053205 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.053121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:45.053259 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.053085    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 00:28:45.053259 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.053246    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 00:28:45.053311 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.053270    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:53.053255698 +0000 UTC m=+47.389510870 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : secret "router-metrics-certs-default" not found
Oct 31 00:28:45.053311 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.053148    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:45.053311 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.053297    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls podName:5266b841-4495-40ff-8277-a8c7b6fdba80 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:53.053282657 +0000 UTC m=+47.389537830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9h6hk" (UID: "5266b841-4495-40ff-8277-a8c7b6fdba80") : secret "samples-operator-tls" not found
Oct 31 00:28:45.053416 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.053315    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls podName:54ecc909-648d-4848-9f49-464285458129 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:53.053305693 +0000 UTC m=+47.389560865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r9hjn" (UID: "54ecc909-648d-4848-9f49-464285458129") : secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:45.053416 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.053370    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:53.053362372 +0000 UTC m=+47.389617544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : configmap references non-existent config key: service-ca.crt
Oct 31 00:28:45.153772 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.153741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:45.153932 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.153803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:45.154350 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.154090    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:45.154350 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.154159    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls podName:2a996538-9d06-447f-bf87-e7dbb8f078c6 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:53.154139328 +0000 UTC m=+47.490394517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls") pod "dns-default-grc8r" (UID: "2a996538-9d06-447f-bf87-e7dbb8f078c6") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:45.154350 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.154249    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:45.154350 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.154262    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f66ccc5cb-h7cvd: secret "image-registry-tls" not found
Oct 31 00:28:45.154350 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.154300    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls podName:d6511745-7315-463c-bc25-0e3e1fd6bf42 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:53.154287924 +0000 UTC m=+47.490543110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls") pod "image-registry-7f66ccc5cb-h7cvd" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42") : secret "image-registry-tls" not found
Oct 31 00:28:45.164179 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.164026    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:45.359114 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.359083    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/1.log"
Oct 31 00:28:45.359631 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.359576    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/0.log"
Oct 31 00:28:45.359631 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.359620    2577 generic.go:356] "Generic (PLEG): container finished" podID="c44bba77-c0b2-49cc-92c1-836cf8013950" containerID="1010115f821e74be8029255505b4165efa642f8bce3ec324faae93519f703f55" exitCode=255
Oct 31 00:28:45.359801 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.359703    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" event={"ID":"c44bba77-c0b2-49cc-92c1-836cf8013950","Type":"ContainerDied","Data":"1010115f821e74be8029255505b4165efa642f8bce3ec324faae93519f703f55"}
Oct 31 00:28:45.359853 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.359796    2577 scope.go:122] "RemoveContainer" containerID="8cd13f8b0ba42b53f2ef0b451a24b706756e3fbbd95c886842779d7291bf956c"
Oct 31 00:28:45.360091 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.360070    2577 scope.go:122] "RemoveContainer" containerID="1010115f821e74be8029255505b4165efa642f8bce3ec324faae93519f703f55"
Oct 31 00:28:45.360313 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:45.360294    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcbjl_openshift-console-operator(c44bba77-c0b2-49cc-92c1-836cf8013950)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" podUID="c44bba77-c0b2-49cc-92c1-836cf8013950"
Oct 31 00:28:45.361708 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.361679    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4sch4" event={"ID":"f53b8d98-bbb7-4108-a65d-6a15ab782bee","Type":"ContainerStarted","Data":"b81900eec2c2405e13a75b226b2fd7840b8cabb06ac2c5be5d7e034aac4b28e6"}
Oct 31 00:28:45.417619 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.417582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:45.615719 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.615229    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-4sch4" podStartSLOduration=292.503569179 podStartE2EDuration="4m59.615209243s" totalImagesPullingTime="7.111640064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:45.613293005 +0000 UTC m=+39.949548201" watchObservedRunningTime="2026-10-31 00:28:45.615209243 +0000 UTC m=+39.951464439"
Oct 31 00:28:45.625756 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.625724    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:45.819442 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.819393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:45.889452 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.889369    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2"]
Oct 31 00:28:45.912011 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.911976    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr"]
Oct 31 00:28:45.912201 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.911971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2"
Oct 31 00:28:45.928023 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.927993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr"
Oct 31 00:28:45.950073 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.948301    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2"]
Oct 31 00:28:45.957456 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.957428    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr"]
Oct 31 00:28:45.965304 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.965279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j45q9\" (UniqueName: \"kubernetes.io/projected/c416b494-9d26-466e-b840-103ce32abf45-kube-api-access-j45q9\") pod \"migrator-66df5d496b-5wmv2\" (UID: \"c416b494-9d26-466e-b840-103ce32abf45\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2"
Oct 31 00:28:45.965504 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.965484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b79cl\" (UniqueName: \"kubernetes.io/projected/bc77171d-558b-40c2-aa20-8e45c1693030-kube-api-access-b79cl\") pod \"migrator-66df5d496b-hbzrr\" (UID: \"bc77171d-558b-40c2-aa20-8e45c1693030\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr"
Oct 31 00:28:45.970101 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.970074    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-g8859\""
Oct 31 00:28:45.970224 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.970161    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 00:28:45.970224 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:45.970171    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:46.040490 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.040461    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:46.066195 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.066149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b79cl\" (UniqueName: \"kubernetes.io/projected/bc77171d-558b-40c2-aa20-8e45c1693030-kube-api-access-b79cl\") pod \"migrator-66df5d496b-hbzrr\" (UID: \"bc77171d-558b-40c2-aa20-8e45c1693030\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr"
Oct 31 00:28:46.066384 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.066264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45q9\" (UniqueName: \"kubernetes.io/projected/c416b494-9d26-466e-b840-103ce32abf45-kube-api-access-j45q9\") pod \"migrator-66df5d496b-5wmv2\" (UID: \"c416b494-9d26-466e-b840-103ce32abf45\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2"
Oct 31 00:28:46.133929 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.133897    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45q9\" (UniqueName: \"kubernetes.io/projected/c416b494-9d26-466e-b840-103ce32abf45-kube-api-access-j45q9\") pod \"migrator-66df5d496b-5wmv2\" (UID: \"c416b494-9d26-466e-b840-103ce32abf45\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2"
Oct 31 00:28:46.134103 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.133904    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b79cl\" (UniqueName: \"kubernetes.io/projected/bc77171d-558b-40c2-aa20-8e45c1693030-kube-api-access-b79cl\") pod \"migrator-66df5d496b-hbzrr\" (UID: \"bc77171d-558b-40c2-aa20-8e45c1693030\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr"
Oct 31 00:28:46.235429 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.235346    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2"
Oct 31 00:28:46.254749 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.254717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr"
Oct 31 00:28:46.262063 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.261510    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:46.348141 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.348114    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cn9rf"]
Oct 31 00:28:46.372876 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.372790    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/1.log"
Oct 31 00:28:46.373568 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.373330    2577 scope.go:122] "RemoveContainer" containerID="1010115f821e74be8029255505b4165efa642f8bce3ec324faae93519f703f55"
Oct 31 00:28:46.373568 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:46.373504    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcbjl_openshift-console-operator(c44bba77-c0b2-49cc-92c1-836cf8013950)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" podUID="c44bba77-c0b2-49cc-92c1-836cf8013950"
Oct 31 00:28:46.376907 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.376879    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:46.377022 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.376924    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pdpbv" event={"ID":"d48c64d8-b46d-48e0-830e-d9ec6a92ff06","Type":"ContainerStarted","Data":"7432fe6a7bdeb85c87d254120244a09b704651558865758f7a156634d1019b6c"}
Oct 31 00:28:46.377022 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.376950    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp" event={"ID":"967855d0-8720-446c-a8df-9e2b427533b3","Type":"ContainerStarted","Data":"226718368ee35554e115e3c7aa4ffa5e5c40f693531b87e1fec4dd7a55da8ea5"}
Oct 31 00:28:46.377127 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.377110    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:46.418462 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.418384    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cn9rf"]
Oct 31 00:28:46.437291 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.437255    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zm47m\""
Oct 31 00:28:46.437291 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.437277    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 00:28:46.438164 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.438145    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 00:28:46.438375 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.438327    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 00:28:46.438541 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.438521    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 00:28:46.464305 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.464273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:46.468486 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.468460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-cabundle\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:46.469174 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.468804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zfbv\" (UniqueName: \"kubernetes.io/projected/dc5110ed-77a9-443b-b64f-5309fadadd78-kube-api-access-9zfbv\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:46.469174 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.468884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-key\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:46.493795 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.493754    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pdpbv" podStartSLOduration=34.672568913 podStartE2EDuration="39.493740191s" totalImagesPullingTime="4.821171278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:46.492981633 +0000 UTC m=+40.829236829" watchObservedRunningTime="2026-10-31 00:28:46.493740191 +0000 UTC m=+40.829995384"
Oct 31 00:28:46.569956 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.569910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfbv\" (UniqueName: \"kubernetes.io/projected/dc5110ed-77a9-443b-b64f-5309fadadd78-kube-api-access-9zfbv\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:46.570170 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.569987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-key\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:46.570170 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.570068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-cabundle\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:46.571046 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.571019    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-cabundle\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:46.573108 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.573082    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-key\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:46.635419 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.635384    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2"]
Oct 31 00:28:46.637875 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.637840    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zfbv\" (UniqueName: \"kubernetes.io/projected/dc5110ed-77a9-443b-b64f-5309fadadd78-kube-api-access-9zfbv\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:46.658607 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:46.658568    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc416b494_9d26_466e_b840_103ce32abf45.slice/crio-337dfcfdc8d1c82b3adb3ba262fd22dc138a1ab7efebe2564a7b62f304372df6 WatchSource:0}: Error finding container 337dfcfdc8d1c82b3adb3ba262fd22dc138a1ab7efebe2564a7b62f304372df6: Status 404 returned error can't find the container with id 337dfcfdc8d1c82b3adb3ba262fd22dc138a1ab7efebe2564a7b62f304372df6
Oct 31 00:28:46.663214 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.663163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:46.684978 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.684946    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr"]
Oct 31 00:28:46.688867 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:46.688832    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc77171d_558b_40c2_aa20_8e45c1693030.slice/crio-b228b0c3a72c8823a4d9fe3cf0b32c79d98829f5a54434df11b5a81dda3abf34 WatchSource:0}: Error finding container b228b0c3a72c8823a4d9fe3cf0b32c79d98829f5a54434df11b5a81dda3abf34: Status 404 returned error can't find the container with id b228b0c3a72c8823a4d9fe3cf0b32c79d98829f5a54434df11b5a81dda3abf34
Oct 31 00:28:46.703128 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.703089    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x7l7z"]
Oct 31 00:28:46.706231 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.706210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.707761 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.707739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:46.762585 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.762422    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:28:46.762585 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.762548    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wzhm5\""
Oct 31 00:28:46.762833 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.762794    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:28:46.774405 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.774314    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x7l7z"]
Oct 31 00:28:46.774405 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.774367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.774618 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.774433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.774618 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.774496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-crio-socket\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.774721 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.774637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjz6q\" (UniqueName: \"kubernetes.io/projected/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-api-access-wjz6q\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.774721 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.774700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-data-volume\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.854615 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.854542    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp" podStartSLOduration=286.571959918 podStartE2EDuration="4m53.854523804s" totalImagesPullingTime="7.282563886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:46.853096472 +0000 UTC m=+41.189351691" watchObservedRunningTime="2026-10-31 00:28:46.854523804 +0000 UTC m=+41.190778997"
Oct 31 00:28:46.876052 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.876003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-crio-socket\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.876198 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.876120    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-crio-socket\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.876198 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.876154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjz6q\" (UniqueName: \"kubernetes.io/projected/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-api-access-wjz6q\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.876286 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.876200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-data-volume\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.876286 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.876238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.876286 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.876276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.876417 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:46.876366    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:46.876467 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:46.876419    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls podName:c83cbbe5-3d68-4067-a0e9-c38f0f72df38 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:47.376401416 +0000 UTC m=+41.712656605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7l7z" (UID: "c83cbbe5-3d68-4067-a0e9-c38f0f72df38") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:46.876545 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:46.876516    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:28:46.876594 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:46.876560    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm podName:c83cbbe5-3d68-4067-a0e9-c38f0f72df38 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:47.37654421 +0000 UTC m=+41.712799395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7l7z" (UID: "c83cbbe5-3d68-4067-a0e9-c38f0f72df38") : configmap "kube-rbac-proxy" not found
Oct 31 00:28:46.876644 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.876617    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-data-volume\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.943175 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.943141    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjz6q\" (UniqueName: \"kubernetes.io/projected/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-api-access-wjz6q\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:46.958083 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:46.958000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:47.084152 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.084072    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cn9rf"]
Oct 31 00:28:47.087636 ip-10-0-132-125 kubenswrapper[2577]: W1031 00:28:47.087595    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5110ed_77a9_443b_b64f_5309fadadd78.slice/crio-be45bfd55cd4bd368bcfed94aa7ff5dbfc89f138e2a7eb7f6d79e2ca58ace1d7 WatchSource:0}: Error finding container be45bfd55cd4bd368bcfed94aa7ff5dbfc89f138e2a7eb7f6d79e2ca58ace1d7: Status 404 returned error can't find the container with id be45bfd55cd4bd368bcfed94aa7ff5dbfc89f138e2a7eb7f6d79e2ca58ace1d7
Oct 31 00:28:47.151264 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.151196    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:47.378804 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.378723    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf" event={"ID":"dc5110ed-77a9-443b-b64f-5309fadadd78","Type":"ContainerStarted","Data":"22eb36a62b24bef9cc738d19b80ce636a2875285e89cfa054d754698530974d1"}
Oct 31 00:28:47.378804 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.378782    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf" event={"ID":"dc5110ed-77a9-443b-b64f-5309fadadd78","Type":"ContainerStarted","Data":"be45bfd55cd4bd368bcfed94aa7ff5dbfc89f138e2a7eb7f6d79e2ca58ace1d7"}
Oct 31 00:28:47.380830 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.380476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:47.380830 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.380485    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr" event={"ID":"bc77171d-558b-40c2-aa20-8e45c1693030","Type":"ContainerStarted","Data":"b228b0c3a72c8823a4d9fe3cf0b32c79d98829f5a54434df11b5a81dda3abf34"}
Oct 31 00:28:47.380830 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:47.380572    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:47.380830 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.380697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:47.380830 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:47.380716    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls podName:c83cbbe5-3d68-4067-a0e9-c38f0f72df38 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:48.380699819 +0000 UTC m=+42.716954991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7l7z" (UID: "c83cbbe5-3d68-4067-a0e9-c38f0f72df38") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:47.380830 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:47.380749    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:28:47.380830 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:47.380808    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm podName:c83cbbe5-3d68-4067-a0e9-c38f0f72df38 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:48.380789384 +0000 UTC m=+42.717044572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7l7z" (UID: "c83cbbe5-3d68-4067-a0e9-c38f0f72df38") : configmap "kube-rbac-proxy" not found
Oct 31 00:28:47.382747 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.382343    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2" event={"ID":"c416b494-9d26-466e-b840-103ce32abf45","Type":"ContainerStarted","Data":"337dfcfdc8d1c82b3adb3ba262fd22dc138a1ab7efebe2564a7b62f304372df6"}
Oct 31 00:28:47.490334 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.490307    2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:47.490471 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.490369    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:47.490698 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.490683    2577 scope.go:122] "RemoveContainer" containerID="1010115f821e74be8029255505b4165efa642f8bce3ec324faae93519f703f55"
Oct 31 00:28:47.490873 ip-10-0-132-125 kubenswrapper[2577]: E1031 00:28:47.490854    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcbjl_openshift-console-operator(c44bba77-c0b2-49cc-92c1-836cf8013950)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" podUID="c44bba77-c0b2-49cc-92c1-836cf8013950"
Oct 31 00:28:47.498083 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.498013    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:47.500770 ip-10-0-132-125 kubenswrapper[2577]: I1031 00:28:47.500029    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf" podStartSLOduration=1.5000108239999999 podStartE2EDuration="1.500010824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:47.49848717 +0000 UTC m=+41.834742364" watchObservedRunningTime="2026-10-31 00:28:47.500010824 +0000 UTC m=+41.836266021"
Oct 31 00:28:47.751057 ip-10-0-132-125 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 00:28:47.765534 ip-10-0-132-125 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 00:28:47.765729 ip-10-0-132-125 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 00:28:47.765789 ip-10-0-132-125 systemd[1]: kubelet.service: Consumed 3.024s CPU time.
Oct 31 00:28:47.782761 ip-10-0-132-125 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:28:47.848108 ip-10-0-132-125 kubenswrapper[4935]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:47.848108 ip-10-0-132-125 kubenswrapper[4935]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:28:47.848108 ip-10-0-132-125 kubenswrapper[4935]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:47.848108 ip-10-0-132-125 kubenswrapper[4935]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:47.850136 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850049    4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:47.850136 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850135    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:47.850136 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850139    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850142    4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850145    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850149    4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850153    4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850160    4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850165    4935 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850169    4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850174    4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850177    4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850181    4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850186    4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850188    4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850191    4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850193    4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850196    4935 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850199    4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850201    4935 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850204    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:47.850241 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850207    4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850209    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850212    4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850214    4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850217    4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850224    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850227    4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850230    4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850233    4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850236    4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850238    4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850241    4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850243    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850246    4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850249    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850252    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850255    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850257    4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850260    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:47.850689 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850262    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850265    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850268    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850271    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850273    4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850276    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850278    4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850281    4935 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850285    4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850288    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850291    4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850294    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850296    4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850299    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850303    4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850305    4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850308    4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850310    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850313    4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850316    4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:47.851200 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850319    4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850322    4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850325    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850327    4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850330    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850332    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850335    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850338    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850340    4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850343    4935 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850345    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850348    4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850350    4935 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850354    4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850357    4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850359    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850362    4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850364    4935 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850367    4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850369    4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:47.851675 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850372    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850375    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850377    4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850380    4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850382    4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850385    4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850387    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850391    4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850393    4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850396    4935 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850398    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850401    4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850404    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850407    4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850410    4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850413    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850415    4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850418    4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850420    4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:47.852164 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850423    4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850426    4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850428    4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850430    4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850433    4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850437    4935 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.850440    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851247    4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851256    4935 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851260    4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851263    4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851266    4935 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851269    4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851272    4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851275    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851277    4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851280    4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851282    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851285    4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851288    4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:47.852616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851291    4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851294    4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851296    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851299    4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851303    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851305    4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851308    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851311    4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851314    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851317    4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851319    4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851322    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851324    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851327    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851330    4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851332    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851335    4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851337    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851340    4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:47.853119 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851342    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851345    4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851347    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851350    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851352    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851355    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851357    4935 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851360    4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851362    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851365    4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851368    4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851370    4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851373    4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851376    4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851379    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851382    4935 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851384    4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851387    4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851389    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851392    4935 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:47.853581 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851395    4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851398    4935 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851400    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851403    4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851405    4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851408    4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851410    4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851412    4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851415    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851417    4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851420    4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851423    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851425    4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851427    4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851430    4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851432    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851435    4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851438    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851440    4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851443    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:47.854071 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851447    4935 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851449    4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851452    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851455    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851458    4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851462    4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851465    4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851468    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851471    4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851473    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851476    4935 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851478    4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851481    4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851485    4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851489    4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851493    4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851495    4935 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851498    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:47.854545 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851501    4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851504    4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851506    4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851509    4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851511    4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851514    4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851516    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851519    4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851521    4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851527    4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851529    4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851532    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851534    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851537    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851540    4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.851543    4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852052    4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852059    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852063    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:47.854993 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852068    4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852070    4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852073    4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852076    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852078    4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852081    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852083    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852086    4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852089    4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852092    4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852095    4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852097    4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852100    4935 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852102    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852105    4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852107    4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852110    4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852113    4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852116    4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852118    4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:47.855486 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852121    4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852123    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852128    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852131    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852133    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852136    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852138    4935 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852141    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852143    4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852146    4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852153    4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852157    4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852160    4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852163    4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852165    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852168    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852171    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852173    4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852176    4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:47.855965 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852179    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852181    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852184    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852187    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852189    4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852192    4935 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852194    4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852197    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852199    4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852202    4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852204    4935 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852207    4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852209    4935 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852212    4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852214    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852218    4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852220    4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852223    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852225    4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:47.856774 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852228    4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852232    4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852235    4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852238    4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852241    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852244    4935 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852246    4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852249    4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852251    4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852254    4935 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852256    4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852259    4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852261    4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852263    4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852266    4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852268    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852271    4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852273    4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852276    4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852278    4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:47.857659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852280    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852283    4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852285    4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852288    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852290    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852293    4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852295    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852297    4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852300    4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852303    4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852306    4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852308    4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852311    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852314    4935 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852316    4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852319    4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852322    4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852325    4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852327    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852330    4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:47.858626 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852332    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852335    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852338    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852340    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.852343    4935 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852446    4935 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852457    4935 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852466    4935 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852473    4935 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852479    4935 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852482    4935 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852487    4935 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852491    4935 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852495    4935 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852497    4935 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852501    4935 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852504    4935 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852508    4935 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852511    4935 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852514    4935 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852517    4935 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852519    4935 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852522    4935 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:28:47.859834 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852526    4935 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852529    4935 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852532    4935 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852535    4935 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852538    4935 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852543    4935 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852546    4935 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852549    4935 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852552    4935 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852555    4935 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852558    4935 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852561    4935 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852564    4935 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852567    4935 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852571    4935 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852574    4935 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852577    4935 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852580    4935 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852583    4935 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852586    4935 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852590    4935 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852593    4935 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852596    4935 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852599    4935 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:28:47.860794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852602    4935 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852606    4935 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852609    4935 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852612    4935 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852615    4935 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852618    4935 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852621    4935 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852624    4935 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852627    4935 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852630    4935 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852632    4935 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852635    4935 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852639    4935 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852643    4935 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852646    4935 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852649    4935 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852652    4935 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852655    4935 flags.go:64] FLAG: --help="false"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852657    4935 flags.go:64] FLAG: --hostname-override="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852661    4935 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852663    4935 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852666    4935 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852669    4935 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:28:47.861833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852673    4935 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852678    4935 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852680    4935 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852683    4935 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852686    4935 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852689    4935 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852692    4935 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852695    4935 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852698    4935 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852701    4935 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852704    4935 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852707    4935 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852710    4935 flags.go:64] FLAG: --lock-file=""
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852712    4935 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852715    4935 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852718    4935 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852724    4935 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852726    4935 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852729    4935 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852732    4935 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852735    4935 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852739    4935 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852742    4935 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852745    4935 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852750    4935 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:28:47.862757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852753    4935 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852757    4935 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852760    4935 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852763    4935 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852765    4935 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852768    4935 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852771    4935 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852774    4935 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852778    4935 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852786    4935 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852789    4935 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852792    4935 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852795    4935 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852798    4935 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852801    4935 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852804    4935 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852807    4935 flags.go:64] FLAG: --port="10250"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852810    4935 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852813    4935 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0eaf5b5e530924e56"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852817    4935 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852819    4935 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852822    4935 flags.go:64] FLAG: --register-node="true"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852825    4935 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852829    4935 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852832    4935 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852834    4935 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:28:47.863747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852838    4935 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852850    4935 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852855    4935 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852858    4935 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852861    4935 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852864    4935 flags.go:64] FLAG: --runonce="false"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852866    4935 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852870    4935 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852872    4935 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852875    4935 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852878    4935 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852881    4935 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852884    4935 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852888    4935 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852892    4935 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852895    4935 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852897    4935 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852900    4935 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852903    4935 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852906    4935 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852909    4935 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852914    4935 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852917    4935 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852920    4935 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852923    4935 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:28:47.864781 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852926    4935 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852929    4935 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852932    4935 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852935    4935 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852937    4935 flags.go:64] FLAG: --v="2"
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852941    4935 flags.go:64] FLAG: --version="false"
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852945    4935 flags.go:64] FLAG: --vmodule=""
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852949    4935 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.852952    4935 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853101    4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853104    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853107    4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853110    4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853113    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853116    4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853118    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853121    4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853124    4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853126    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853129    4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853131    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853136    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:47.865792 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853138    4935 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853141    4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853143    4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853146    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853148    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853151    4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853154    4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853156    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853159    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853161    4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853164    4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853167    4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853169    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853172    4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853174    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853177    4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853179    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853182    4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853184    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853189    4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:47.866659 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853191    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853194    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853196    4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853199    4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853201    4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853203    4935 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853206    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853208    4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853211    4935 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853213    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853216    4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853220    4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853223    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853225    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853228    4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853230    4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853232    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853235    4935 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853237    4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853240    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:47.867465 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853243    4935 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853245    4935 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853247    4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853250    4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853252    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853255    4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853257    4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853260    4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853263    4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853265    4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853267    4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853271    4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853273    4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853276    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853278    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853281    4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853284    4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853286    4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853289    4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:47.868282 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853291    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853294    4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853296    4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853299    4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853302    4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853305    4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853307    4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853310    4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853312    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853315    4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853317    4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853319    4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853322    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853325    4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853327    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853330    4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853332    4935 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853335    4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853337    4935 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:47.869261 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853340    4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853342    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853345    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853348    4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853350    4935 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853354    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853356    4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853358    4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853362    4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853366    4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853368    4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853371    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853375    4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853379    4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853382    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:47.870004 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.853408    4935 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853443    4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853448    4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853452    4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853455    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853459    4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853462    4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853465    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853468    4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853470    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853473    4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853475    4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853479    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853481    4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853484    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853486    4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853489    4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853491    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853494    4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:47.870598 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853496    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853499    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853502    4935 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853505    4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853508    4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853510    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853513    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853515    4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853517    4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853520    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853522    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853525    4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853527    4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853530    4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853532    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853534    4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853538    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853540    4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853543    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853545    4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:47.871347 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853548    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853551    4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853553    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853556    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853558    4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853560    4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853571    4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853573    4935 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853576    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853578    4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853581    4935 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853583    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853586    4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853589    4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853591    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853596    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853598    4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853601    4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853603    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853606    4935 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:47.872146 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853608    4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853611    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853613    4935 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853616    4935 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853618    4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853621    4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853623    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853626    4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853630    4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853633    4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853636    4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853638    4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853641    4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853644    4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853646    4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853649    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853651    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853654    4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853656    4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853659    4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:47.872924 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853662    4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853665    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853667    4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853670    4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853672    4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853675    4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853677    4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853681    4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853684    4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853686    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853689    4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853691    4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853694    4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853696    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853699    4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853701    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853704    4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853706    4935 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853708    4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:47.873710 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853711    4935 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853715    4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853718    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853720    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853724    4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853726    4935 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853729    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853732    4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.853734    4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.859162    4935 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.859183    4935 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.859205    4935 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.859212    4935 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859300    4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859308    4935 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859313    4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859317    4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859322    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859327    4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859331    4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:47.874447 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859335    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859339    4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859343    4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859347    4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859350    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859355    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859359    4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859363    4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859368    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859372    4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859377    4935 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859380    4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859384    4935 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859388    4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859392    4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859396    4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859400    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859404    4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859408    4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859412    4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:47.875256 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859417    4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859420    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859434    4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859439    4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859443    4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859447    4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859451    4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859456    4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859460    4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859464    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859468    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859472    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859477    4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859481    4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859485    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859489    4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859493    4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859500    4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859506    4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:47.876052 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859511    4935 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859516    4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859520    4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859524    4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859528    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859532    4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859537    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859541    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859545    4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859549    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859555    4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859562    4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859566    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859571    4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859575    4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859579    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859592    4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859597    4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859601    4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:47.876797 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859606    4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859610    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859615    4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859619    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859623    4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859627    4935 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859631    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859635    4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859639    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859643    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859647    4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859651    4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859655    4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859659    4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859664    4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859668    4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859680    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859685    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859689    4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:47.877740 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859693    4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859697    4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859701    4935 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859705    4935 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859710    4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859714    4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859718    4935 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859722    4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859726    4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859730    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859734    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859746    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859751    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859754    4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859758    4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859763    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859767    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859771    4935 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859775    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859779    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:47.878493 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859783    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.859787    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.859829    4935 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861132    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861149    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861152    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861155    4935 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861159    4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861162    4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861165    4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861172    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861174    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861177    4935 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861180    4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861183    4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861185    4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:47.879275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861189    4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861196    4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861201    4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861204    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861206    4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861209    4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861214    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861217    4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861220    4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861223    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861226    4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861229    4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861232    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861235    4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861237    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861240    4935 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861242    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861245    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861248    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861253    4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:47.879899 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861257    4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861260    4935 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861263    4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861265    4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861268    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861271    4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861274    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861276    4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861279    4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861282    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861285    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861287    4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861292    4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861295    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861298    4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861300    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861303    4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861305    4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861308    4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:47.880699 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861310    4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861313    4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861316    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861320    4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861323    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861329    4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861332    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861335    4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861338    4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861340    4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861343    4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861346    4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861349    4935 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861351    4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861355    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861358    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861360    4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861363    4935 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861368    4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861371    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:47.881517 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861374    4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861376    4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861379    4935 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861381    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861384    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861387    4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861389    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861392    4935 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861395    4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861398    4935 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861403    4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861405    4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861408    4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861411    4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861414    4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861416    4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861419    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861421    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861424    4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861427    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:47.882318 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861430    4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861432    4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861435    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861440    4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861442    4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861446    4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861448    4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861452    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861454    4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861457    4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861459    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861462    4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861465    4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861467    4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.861506    4935 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:47.882987 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861601    4935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861606    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861609    4935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861613    4935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861615    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861618    4935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861621    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861624    4935 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861626    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861629    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861635    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861637    4935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861640    4935 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861643    4935 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861646    4935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861648    4935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861651    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861654    4935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861657    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:47.883568 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861659    4935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861662    4935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861665    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861667    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861672    4935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861675    4935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861678    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861681    4935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861684    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861686    4935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861689    4935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861691    4935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861694    4935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861696    4935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861699    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861703    4935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861706    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861712    4935 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861714    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:47.884403 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861717    4935 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861719    4935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861722    4935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861725    4935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861727    4935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861730    4935 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861733    4935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861736    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861738    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861741    4935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861746    4935 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861748    4935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861751    4935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861754    4935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861756    4935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861759    4935 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861762    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861764    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861767    4935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861770    4935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:47.884974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861773    4935 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861776    4935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861781    4935 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861784    4935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861786    4935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861789    4935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861792    4935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861794    4935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861797    4935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861799    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861802    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861804    4935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861807    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861810    4935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861812    4935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861817    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861820    4935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861823    4935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861826    4935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:47.885478 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861828    4935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861831    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861834    4935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861837    4935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861839    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861842    4935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861845    4935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861848    4935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861853    4935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861856    4935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861858    4935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861861    4935 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861864    4935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861867    4935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861870    4935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861872    4935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861875    4935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861877    4935 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861880    4935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861882    4935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:47.886008 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861885    4935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861892    4935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861895    4935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861898    4935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861901    4935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861904    4935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861906    4935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861909    4935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:47.861911    4935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.861980    4935 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.863398    4935 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.863470    4935 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.863725    4935 server.go:1014] "Starting client certificate rotation"
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.863862    4935 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.863909    4935 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:23:06 +0000 UTC" deadline="2028-04-14 00:07:51.789363197 +0000 UTC"
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.863944    4935 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12743h39m3.92542092s"
Oct 31 00:28:47.886616 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.864200    4935 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:28:47.886999 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.865338    4935 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:28:47.886999 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.866596    4935 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:28:47.886999 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.868260    4935 log.go:25] "Validated CRI v1 image API"
Oct 31 00:28:47.886999 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.869010    4935 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:28:47.886999 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.870930    4935 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 90e9edcb-ed98-4137-b600-e2f6ad30a1eb:/dev/nvme0n1p4 fc2ad222-b3fa-420f-8478-5423111f0e52:/dev/nvme0n1p3]
Oct 31 00:28:47.887201 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.870950    4935 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2236639900fd66a76022f581d57a34b0c3a890430f3ef984b48b96a5cddd6896/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2236639900fd66a76022f581d57a34b0c3a890430f3ef984b48b96a5cddd6896/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24f3f5a6d858460eadc4193742eacb162f14f47f097c451919ae080b66f127bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24f3f5a6d858460eadc4193742eacb162f14f47f097c451919ae080b66f127bd/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/337dfcfdc8d1c82b3adb3ba262fd22dc138a1ab7efebe2564a7b62f304372df6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/337dfcfdc8d1c82b3adb3ba262fd22dc138a1ab7efebe2564a7b62f304372df6/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d4dcfbc57887f3ae26e161794606c8a4eec7b2c4a0a5c246c7b3425f645a1fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d4dcfbc57887f3ae26e161794606c8a4eec7b2c4a0a5c246c7b3425f645a1fa/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4045b6bbec4c2576d027ccc451a90f03228f3a12d53680353956648ad43643c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4045b6bbec4c2576d027ccc451a90f03228f3a12d53680353956648ad43643c6/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4479da8f52de7c7e460cf7dd7e12df5a099ceb3a815f3d483c29d4c3781924d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4479da8f52de7c7e460cf7dd7e12df5a099ceb3a815f3d483c29d4c3781924d5/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f91ec2ee852545672bc9664a804845aaaac3a56dd03e60e633125fd790c0099/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f91ec2ee852545672bc9664a804845aaaac3a56dd03e60e633125fd790c0099/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5037dff5fe581117ec599ee6a185baa4519a23dad1d40dfda65ed4c47a78c1a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5037dff5fe581117ec599ee6a185baa4519a23dad1d40dfda65ed4c47a78c1a4/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59d770ac32b29db330d9d03ff2a4e039a22e00206091186d6c4afbeeefce25ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59d770ac32b29db330d9d03ff2a4e039a22e00206091186d6c4afbeeefce25ef/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ca98829f18eb3109da7316310c5cdb18d632f9adabb09daa11c186b13077209/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ca98829f18eb3109da7316310c5cdb18d632f9adabb09daa11c186b13077209/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ce410fcc02e43610380bb82c65b50f64af5c4d76a4d54c987c22b45f2df187e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ce410fcc02e43610380bb82c65b50f64af5c4d76a4d54c987c22b45f2df187e/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d7183f126a97fedb177bb2869ecb7eeebb9b8194d8d3e2c2075bb5d11fe866f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d7183f126a97fedb177bb2869ecb7eeebb9b8194d8d3e2c2075bb5d11fe866f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91827ae633ef0d4dc544a5124cff2d6c01b8156f9728bea844a8ad5713880a61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91827ae633ef0d4dc544a5124cff2d6c01b8156f9728bea844a8ad5713880a61/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92169b77fc7554447d2dd9fb89c3b6c9fb184a3190082a40f1f251f1d1e0588f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92169b77fc7554447d2dd9fb89c3b6c9fb184a3190082a40f1f251f1d1e0588f/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99cd239f879c4d8945493c1c5fd1bbbc9d46e942c27f545275e820265d06f214/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99cd239f879c4d8945493c1c5fd1bbbc9d46e942c27f545275e820265d06f214/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a12afa6f7ebfd8eeaba6e68fc9759bfb06eb34678d2cd2af4a1a9c44a43fe539/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a12afa6f7ebfd8eeaba6e68fc9759bfb06eb34678d2cd2af4a1a9c44a43fe539/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7aea42d2a498166e52d91964f650a8c131a6483cf0a77cd63764dc7f9651079/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7aea42d2a498166e52d91964f650a8c131a6483cf0a77cd63764dc7f9651079/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad0ab66cdd8de4cb2648cfa7d40b02409fb1fd42d5fba5885a60caadaa54d068/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad0ab66cdd8de4cb2648cfa7d40b02409fb1fd42d5fba5885a60caadaa54d068/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b228b0c3a72c8823a4d9fe3cf0b32c79d98829f5a54434df11b5a81dda3abf34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b228b0c3a72c8823a4d9fe3cf0b32c79d98829f5a54434df11b5a81dda3abf34/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be45bfd55cd4bd368bcfed94aa7ff5dbfc89f138e2a7eb7f6d79e2ca58ace1d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be45bfd55cd4bd368bcfed94aa7ff5dbfc89f138e2a7eb7f6d79e2ca58ace1d7/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4b98ab84513804906c41697e62ab0a3f72ac8f3870f11868c9f05645281992b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4b98ab84513804906c41697e62ab0a3f72ac8f3870f11868c9f05645281992b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dac2a2c016684a2192de8284e955022ba30f23dc6bcfd2e1938cf517b903bf91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dac2a2c016684a2192de8284e955022ba30f23dc6bcfd2e1938cf517b903bf91/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efc2ad77731ea8e8e9d413d9c176a4569eacfea00892764a8acba158ce419e5f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efc2ad77731ea8e8e9d413d9c176a4569eacfea00892764a8acba158ce419e5f/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f653f495285a89d3f558729b86f578d22e2e440db158445246c22cc032cf67db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f653f495285a89d3f558729b86f578d22e2e440db158445246c22cc032cf67db/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff9b9b39519a0df5ce768b87cff7916719e9a31d4bbe44186736fa34333013c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff9b9b39519a0df5ce768b87cff7916719e9a31d4bbe44186736fa34333013c0/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67064c-ba5d-4780-9257-9df20f304f29/volumes/kubernetes.io~projected/kube-api-access-r4f8k:{mountpoint:/var/lib/kubelet/pods/0e67064c-ba5d-4780-9257-9df20f304f29/volumes/kubernetes.io~projected/kube-api-access-r4f8k major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/261b0474-eb3d-4f3c-b33d-01856306e928/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/261b0474-eb3d-4f3c-b33d-01856306e928/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a996538-9d06-447f-bf87-e7dbb8f078c6/volumes/kubernetes.io~projected/kube-api-access-d7lmn:{mountpoint:/var/lib/kubelet/pods/2a996538-9d06-447f-bf87-e7dbb8f078c6/volumes/kubernetes.io~projected/kube-api-access-d7lmn major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e209316-e440-4bdf-b2d9-cc79ac3e670f/volumes/kubernetes.io~projected/kube-api-access-tq9rt:{mountpoint:/var/lib/kubelet/pods/2e209316-e440-4bdf-b2d9-cc79ac3e670f/volumes/kubernetes.io~projected/kube-api-access-tq9rt major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e209316-e440-4bdf-b2d9-cc79ac3e670f/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/2e209316-e440-4bdf-b2d9-cc79ac3e670f/volumes/kubernetes.io~secret/default-certificate major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e209316-e440-4bdf-b2d9-cc79ac3e670f/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/2e209316-e440-4bdf-b2d9-cc79ac3e670f/volumes/kubernetes.io~secret/stats-auth major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34748730-f975-4008-9a37-dad912c7e930/volumes/kubernetes.io~projected/kube-api-access-7t88q:{mountpoint:/var/lib/kubelet/pods/34748730-f975-4008-9a37-dad912c7e930/volumes/kubernetes.io~projected/kube-api-access-7t88q major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34d3cbf5-134e-4f86-8b33-04ef38cdb685/volumes/kubernetes.io~projected/kube-api-access-2q9pr:{mountpoint:/var/lib/kubelet/pods/34d3cbf5-134e-4f86-8b33-04ef38cdb685/volumes/kubernetes.io~projected/kube-api-access-2q9pr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/408faab9-57a4-4c18-86ae-68772011de21/volumes/kubernetes.io~projected/kube-api-access-mdphh:{mountpoint:/var/lib/kubelet/pods/408faab9-57a4-4c18-86ae-68772011de21/volumes/kubernetes.io~projected/kube-api-access-mdphh major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5266b841-4495-40ff-8277-a8c7b6fdba80/volumes/kubernetes.io~projected/kube-api-access-n42j6:{mountpoint:/var/lib/kubelet/pods/5266b841-4495-40ff-8277-a8c7b6fdba80/volumes/kubernetes.io~projected/kube-api-access-n42j6 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ecc909-648d-4848-9f49-464285458129/volumes/kubernetes.io~projected/kube-api-access-2d5cn:{mountpoint:/var/lib/kubelet/pods/54ecc909-648d-4848-9f49-464285458129/volumes/kubernetes.io~projected/kube-api-access-2d5cn major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/693a3c1f-8228-4691-bd99-f05fdf784cd4/volumes/kubernetes.io~projected/kube-api-access-5tz7l:{mountpoint:/var/lib/kubelet/pods/693a3c1f-8228-4691-bd99-f05fdf784cd4/volumes/kubernetes.io~projected/kube-api-access-5tz7l major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~projected/kube-api-access-xfmz7:{mountpoint:/var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~projected/kube-api-access-xfmz7 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ab38846-5d17-4bf1-92c2-ac823690fc98/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7ab38846-5d17-4bf1-92c2-ac823690fc98/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81be0f35-6496-46d5-a27b-46bd185e3993/volumes/kubernetes.io~projected/kube-api-access-jb2gk:{mountpoint:/var/lib/kubelet/pods/81be0f35-6496-46d5-a27b-46bd185e3993/volumes/kubernetes.io~projected/kube-api-access-jb2gk major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/967855d0-8720-446c-a8df-9e2b427533b3/volumes/kubernetes.io~projected/kube-api-access-czwhh:{mountpoint:/var/lib/kubelet/pods/967855d0-8720-446c-a8df-9e2b427533b3/volumes/kubernetes.io~projected/kube-api-access-czwhh major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/990975fd-2466-4224-acbe-6269e49a9ef9/volumes/kubernetes.io~projected/kube-api-access-cbmnh:{mountpoint:/var/lib/kubelet/pods/990975fd-2466-4224-acbe-6269e49a9ef9/volumes/kubernetes.io~projected/kube-api-access-cbmnh major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/990975fd-2466-4224-acbe-6269e49a9ef9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/990975fd-2466-4224-acbe-6269e49a9ef9/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a27a3030-72b2-4223-b95e-c3997a8eb0f6/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a27a3030-72b2-4223-b95e-c3997a8eb0f6/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a27a3030-72b2-4223-b95e-c3997a8eb0f6/volumes/kubernetes.io~projected/kube-api-access-86wrt:{mountpoint:/var/lib/kubelet/pods/a27a3030-72b2-4223-b95e-c3997a8eb0f6/volumes/kubernetes.io~projected/kube-api-access-86wrt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a27a3030-72b2-4223-b95e-c3997a8eb0f6/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a27a3030-72b2-4223-b95e-c3997a8eb0f6/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1402f00-5adc-44cb-961c-a2556e51529b/volumes/kubernetes.io~projected/kube-api-access-6lvrb:{mountpoint:/var/lib/kubelet/pods/b1402f00-5adc-44cb-961c-a2556e51529b/volumes/kubernetes.io~projected/kube-api-access-6lvrb major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9fad5d0-0d3a-4ebe-a07b-1899400059a7/volumes/kubernetes.io~projected/kube-api-access-495pb:{mountpoint:/var/lib/kubelet/pods/b9fad5d0-0d3a-4ebe-a07b-1899400059a7/volumes/kubernetes.io~projected/kube-api-access-495pb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc77171d-558b-40c2-aa20-8e45c1693030/volumes/kubernetes.io~projected/kube-api-access-b79cl:{mountpoint:/var/lib/kubelet/pods/bc77171d-558b-40c2-aa20-8e45c1693030/volumes/kubernetes.io~projected/kube-api-access-b79cl major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c416b494-9d26-466e-b840-103ce32abf45/volumes/kubernetes.io~projected/kube-api-access-j45q9:{mountpoint:/var/lib/kubelet/pods/c416b494-9d26-466e-b840-103ce32abf45/volumes/kubernetes.io~projected/kube-api-access-j45q9 major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c44bba77-c0b2-49cc-92c1-836cf8013950/volumes/kubernetes.io~projected/kube-api-access-44dtj:{mountpoint:/var/lib/kubelet/pods/c44bba77-c0b2-49cc-92c1-836cf8013950/volumes/kubernetes.io~projected/kube-api-access-44dtj major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c44bba77-c0b2-49cc-92c1-836cf8013950/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c44bba77-c0b2-49cc-92c1-836cf8013950/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8302492-d696-40ad-bffd-431b35319e22/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c8302492-d696-40ad-bffd-431b35319e22/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8302492-d696-40ad-bffd-431b35319e22/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c8302492-d696-40ad-bffd-431b35319e22/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8302492-d696-40ad-bffd-431b35319e22/volumes/kubernetes.io~projected/kube-api-access-44wtx:{mountpoint:/var/lib/kubelet/pods/c8302492-d696-40ad-bffd-431b35319e22/volumes/kubernetes.io~projected/kube-api-access-44wtx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c83cbbe5-3d68-4067-a0e9-c38f0f72df38/volumes/kubernetes.io~projected/kube-api-access-wjz6q:{mountpoint:/var/lib/kubelet/pods/c83cbbe5-3d68-4067-a0e9-c38f0f72df38/volumes/kubernetes.io~projected/kube-api-access-wjz6q major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d48c64d8-b46d-48e0-830e-d9ec6a92ff06/volumes/kubernetes.io~projected/kube-api-access-l5k6g:{mountpoint:/var/lib/kubelet/pods/d48c64d8-b46d-48e0-830e-d9ec6a92ff06/volumes/kubernetes.io~projected/kube-api-access-l5k6g major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~projected/kube-api-access-9ppwn:{mountpoint:/var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~projected/kube-api-access-9ppwn major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7ef07d0-5483-4176-bfb2-4e03c9be4498/volumes/kubernetes.io~projected/kube-api-access-kfs9t:{mountpoint:/var/lib/kubelet/pods/d7ef07d0-5483-4176-bfb2-4e03c9be4498/volumes/kubernetes.io~projected/kube-api-access-kfs9t major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db65a944-e395-4d70-9807-c061aecf0584/volumes/kubernetes.io~projected/kube-api-access-rpz4f:{mountpoint:/var/lib/kubelet/pods/db65a944-e395-4d70-9807-c061aecf0584/volumes/kubernetes.io~projected/kube-api-access-rpz4f major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db65a944-e395-4d70-9807-c061aecf0584/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/db65a944-e395-4d70-9807-c061aecf0584/volumes/kubernetes.io~secret/cert major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed/volumes/kubernetes.io~projected/kube-api-access-z89wv:{mountpoint:/var/lib/kubelet/pods/dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed/volumes/kubernetes.io~projected/kube-api-access-z89wv major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc5110ed-77a9-443b-b64f-5309fadadd78/volumes/kubernetes.io~projected/kube-api-access-9zfbv:{mountpoint:/var/lib/kubelet/pods/dc5110ed-77a9-443b-b64f-5309fadadd78/volumes/kubernetes.io~projected/kube-api-access-9zfbv major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc5110ed-77a9-443b-b64f-5309fadadd78/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/dc5110ed-77a9-443b-b64f-5309fadadd78/volumes/kubernetes.io~secret/signing-key major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f53b8d98-bbb7-4108-a65d-6a15ab782bee/volumes/kubernetes.io~projected/kube-api-access-dvtlp:{mountpoint:/var/lib/kubelet/pods/f53b8d98-bbb7-4108-a65d-6a15ab782bee/volumes/kubernetes.io~projected/kube-api-access-dvtlp major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f53b8d98-bbb7-4108-a65d-6a15ab782bee/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f53b8d98-bbb7-4108-a65d-6a15ab782bee/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5703958-5751-4c4d-b979-01585c84cd46/volumes/kubernetes.io~projected/kube-api-access-dnvlx:{mountpoint:/var/lib/kubelet/pods/f5703958-5751-4c4d-b979-01585c84cd46/volumes/kubernetes.io~projected/kube-api-access-dnvlx major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f59717aa-c796-4074-b63f-2fc7e13f3633/volumes/kubernetes.io~projected/kube-api-access-5rbld:{mountpoint:/var/lib/kubelet/pods/f59717aa-c796-4074-b63f-2fc7e13f3633/volumes/kubernetes.io~projected/kube-api-access-5rbld major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f59717aa-c796-4074-b63f-2fc7e13f3633/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f59717aa-c796-4074-b63f-2fc7e13f3633/volumes/kubernetes.io~secret/serving-cert major:0 minor:228 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7dd03408bbb63a45faf7a8943b111a0de5ab06971540bfd31b266826ca028701/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fee3b75b634fa1906cc7c9291ed73a6bee30a34c87e1e6dec767acac783aa4dc/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4b46a577ad19373a93bf4fdee57b7da02435aec3420cecfdc685652566da0080/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9b7b2df073bd1e9afa8baaddc6a2e2729cf3b194e6afceada8ea85526a72dca6/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8a218b6d0158b88debc6df9360eee8ed7d21cbff00a2fa49fbe5c4e80e81574b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/dda4517730506e820c30e58a252eaf48bed4041fd1d0cf097a11c1c27f203096/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e4bb068726713a0aed27ef6ce9adda1042fcfbe33b65803e736a966873c490d0/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/23c0ac90addc326d7bea4c8f35cbd42c5da138e1bf9ab9842178f79338ec9f81/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2ae0d4d35ba952bb7d5801300dfd9dcb920f467cf1df912899a92e16727ccbff/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/25be5b1527277759033918a52c225a9cef6a18a997601fb0411ada836330928e/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/a7fa89e484e146465c45dd62caf4bcac240ce128314a921587b70b83aeec5214/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/3251cd1e761cba71705d6b3c3ff741329b3c0c8fad141843086b02eedef70166/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/de92d6fd6f1fc137eee4e6b82464f47504a59c4e368969fd7dfa6371c9874e24/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/fcd00ae31229f2109aff2599077f056d5dfa117fa57b94a23a7d8cfc0aeec247/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4e53c0303cbd1e4f14d7287397b6118cacb097d3d8e915a089774578bfb7f522/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/92641f8736c4f6ff15cc0a5c191e07a0aee91752f58cf40c11e9f196eb3d470b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/20e741c362037216d8ad9f88e59e5180fda23822b9ce04422feba3d0fbf76f06/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fe6b01df5a7ac3c93d2d154b353f96e3f377e3464e97f5fe82c24c234a3960c4/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4c32cdae7174219fcebd76dbd6c3f0b64beb61770adba5dd414f9db4cd409780/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d2543f06ebb717581a3a89ae88484478e4b878e5aeac56bc8194f8f92f928336/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a47be3d8841f1c89658c4dc4027221cbbbe685069995fe85da7f34a1510e8748/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/ab78408b6e4f09738e17d63cfe7b93d1ac58d14def995a5c8de2418f5810a6a5/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/64e6bc30ce825b3d979ad340f6e95faa7226cd09cb35b6e159f7a856928e0f1e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/185e4f5fb0b2a4ce370c3c88de3029dbbed38a2eb41a8f15098e9f70cd80eb65/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/e73e60dae137db8a53432e5a67dd0d91f7e813eccc61aff4ca7cd08d993c668b/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/ec14c0bedd444ce628ca6543e6b1ed7023576b791d14740898013a9df01c489d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/1894b1b3c374bf1e03bd25fb5224049f5c91df14cfb55a3987fb30a48e267b23/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/14635ae27d152372a8ff7875c4125060399eba14495f112d287f2b95d32918f9/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/43003583064ea941b658802306b42324f15eb5213c8202e8c8ab5b731141c26a/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/6704493fe82b655471b8167355f8f589e804391e353ecd07415e75ae89583190/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/259ddbf305dd416b81a1f41094d92d5ed637c5df7177cdc8424e81a5f0fc573e/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/5a6048eb241155a20ac31a20c95e8980b4933d442d2010e67688d3b77d7b1463/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/e95a4fe1c24e591d9d12569af7f880f4acf142789bd9617b7f77d0ef8d242160/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/57decc40aac349a19f8faccb9e07b93325418f164f6ed1321e908c0223c2fe0d/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/2e0c8500eae490fb065a26e90d09a4ad02eb6e4bb8dc0e42f8f5e0eec39206f4/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/ec627dd4188cd1a6d82783321a0117847deafe99465ec4921afa21bfae74b930/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/fb03acbed95210ed4e100c8663e6e32ec195d1ee0a6ae7290af9dd964d1d2751/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/7cc950f0fef99a6337166c4b77b36cbc7a4f0b2f5f0d61014fc8c9eaf869ada2/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/32a6d8ca5ca0a06a33ac2cc6f31b2f8656696d6d0f3cc45c4ea06de44dcefba7/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/4d9216251360fa74c95f911e60e6f0b885492293ae307909893330b01fd64749/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/0b0209ddd3522c60c4f51982a605124526ab5352973c38694d52a19ca68f5977/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/61b8c14ae792a90c7266b18ad6a8e7f1becb9260ced82de3b65094480fc24ffc/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/717f4a0b4ed744c2e0b38acadd0d4c9d95103e579c315703b04f764161725ed7/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/808c2547551f398224a77f0e23b9b63aef7853cf35f089079f6ad282d3c3e7f1/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/3ac7880e30cbd0fccbf54d4db54b1c0b48c9a2530e45e5bb6f14768f0c1a6737/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/ed405e8053bce70369ac831e7d2d72cefb2f60d4d12abb3f3a462067c1a2818c/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/19fd2c2cca1da0cb54a50be7dc58c36464eb5800916689ee055958cefdcb8597/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/57d78309debdb1f0759135dff9289144c90734c446c16725501fed0c459d65bc/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b250c9ec3aee725831f66dcc432c5d15a49a07ed9171170cbb56c5a14e7245b1/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/15b60c138a5852055be568c48da71b7873c5ff3507f78602f1fb06e962462dba/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6a8f3d692d8d74e80dcffd3e2f3d1f1df181ac1a93db3860192dc94645a606e9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/78576b88af5773257a565f37b56e3c89ca898359997cdbc785a9557cfd425d89/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/5d11a1419ec5151e52d671d3fe96aa37e5d271cd21a8ef46ca6f3ff6581db1b6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f7df188971bf0993fba964f7934c9edff9cca9a0e254a5fbf8f46be9c7230502/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9c094b2d53fe5f2596dad24188c822a2b9a21b2138654f9a45612bcf1fea5dc2/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/206b74ca6aae24a1b47f9dda8759414d4d7efda50d277840be17e4f84f374cd5/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/26803c7926384075590f2e9065e7320fdfe725ce7718eadac36d9e612e7fcd72/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e93aa216322b722fb817c7ddd3125f5ca68cd62d44dc2e9fac47a5b99d7233ee/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.880242    4935 manager.go:217] Machine: {Timestamp:2026-10-31 00:28:47.879711764 +0000 UTC m=+0.062241051 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3189908 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22b9b7f62885fbdcc758ad0b7e8384 SystemUUID:ec22b9b7-f628-85fb-dcc7-58ad0b7e8384 BootID:81186f95-cfea-4de4-816b-d40bdc47eced Filesystems:[{Device:/var/lib/kubelet/pods/bc77171d-558b-40c2-aa20-8e45c1693030/volumes/kubernetes.io~projected/kube-api-access-b79cl DeviceMajor:0 DeviceMinor:334 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be45bfd55cd4bd368bcfed94aa7ff5dbfc89f138e2a7eb7f6d79e2ca58ace1d7/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b9fad5d0-0d3a-4ebe-a07b-1899400059a7/volumes/kubernetes.io~projected/kube-api-access-495pb DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91827ae633ef0d4dc544a5124cff2d6c01b8156f9728bea844a8ad5713880a61/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4045b6bbec4c2576d027ccc451a90f03228f3a12d53680353956648ad43643c6/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ce410fcc02e43610380bb82c65b50f64af5c4d76a4d54c987c22b45f2df187e/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34d3cbf5-134e-4f86-8b33-04ef38cdb685/volumes/kubernetes.io~projected/kube-api-access-2q9pr DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ca98829f18eb3109da7316310c5cdb18d632f9adabb09daa11c186b13077209/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dac2a2c016684a2192de8284e955022ba30f23dc6bcfd2e1938cf517b903bf91/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db65a944-e395-4d70-9807-c061aecf0584/volumes/kubernetes.io~projected/kube-api-access-rpz4f DeviceMajor:0 DeviceMinor:238 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/261b0474-eb3d-4f3c-b33d-01856306e928/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f5703958-5751-4c4d-b979-01585c84cd46/volumes/kubernetes.io~projected/kube-api-access-dnvlx DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~projected/kube-api-access-xfmz7 DeviceMajor:0 DeviceMinor:207 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67064c-ba5d-4780-9257-9df20f304f29/volumes/kubernetes.io~projected/kube-api-access-r4f8k DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff9b9b39519a0df5ce768b87cff7916719e9a31d4bbe44186736fa34333013c0/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/990975fd-2466-4224-acbe-6269e49a9ef9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54ecc909-648d-4848-9f49-464285458129/volumes/kubernetes.io~projected/kube-api-access-2d5cn DeviceMajor:0 DeviceMinor:217 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f59717aa-c796-4074-b63f-2fc7e13f3633/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:228 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a27a3030-72b2-4223-b95e-c3997a8eb0f6/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/2a996538-9d06-447f-bf87-e7dbb8f078c6/volumes/kubernetes.io~projected/kube-api-access-d7lmn DeviceMajor:0 DeviceMinor:240 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b228b0c3a72c8823a4d9fe3cf0b32c79d98829f5a54434df11b5a81dda3abf34/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ab38846-5d17-4bf1-92c2-ac823690fc98/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5037dff5fe581117ec599ee6a185baa4519a23dad1d40dfda65ed4c47a78c1a4/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a27a3030-72b2-4223-b95e-c3997a8eb0f6/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c8302492-d696-40ad-bffd-431b35319e22/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/81be0f35-6496-46d5-a27b-46bd185e3993/volumes/kubernetes.io~projected/kube-api-access-jb2gk DeviceMajor:0 DeviceMinor:208 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a27a3030-72b2-4223-b95e-c3997a8eb0f6/volumes/kubernetes.io~projected/kube-api-access-86wrt DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59d770ac32b29db330d9d03ff2a4e039a22e00206091186d6c4afbeeefce25ef/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2e209316-e440-4bdf-b2d9-cc79ac3e670f/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:212 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~projected/kube-api-access-9ppwn DeviceMajor:0 DeviceMinor:241 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/b1402f00-5adc-44cb-961c-a2556e51529b/volumes/kubernetes.io~projected/kube-api-access-6lvrb DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c8302492-d696-40ad-bffd-431b35319e22/volumes/kubernetes.io~projected/kube-api-access-44wtx DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efc2ad77731ea8e8e9d413d9c176a4569eacfea00892764a8acba158ce419e5f/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/990975fd-2466-4224-acbe-6269e49a9ef9/volumes/kubernetes.io~projected/kube-api-access-cbmnh DeviceMajor:0 DeviceMinor:216 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:237 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d48c64d8-b46d-48e0-830e-d9ec6a92ff06/volumes/kubernetes.io~projected/kube-api-access-l5k6g DeviceMajor:0 DeviceMinor:289 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d7ef07d0-5483-4176-bfb2-4e03c9be4498/volumes/kubernetes.io~projected/kube-api-access-kfs9t DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7aea42d2a498166e52d91964f650a8c131a6483cf0a77cd63764dc7f9651079/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c44bba77-c0b2-49cc-92c1-836cf8013950/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/967855d0-8720-446c-a8df-9e2b427533b3/volumes/kubernetes.io~projected/kube-api-access-czwhh DeviceMajor:0 DeviceMinor:218 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad0ab66cdd8de4cb2648cfa7d40b02409fb1fd42d5fba5885a60caadaa54d068/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f91ec2ee852545672bc9664a804845aaaac3a56dd03e60e633125fd790c0099/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/337dfcfdc8d1c82b3adb3ba262fd22dc138a1ab7efebe2564a7b62f304372df6/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/693a3c1f-8228-4691-bd99-f05fdf784cd4/volumes/kubernetes.io~projected/kube-api-access-5tz7l DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f653f495285a89d3f558729b86f578d22e2e440db158445246c22cc032cf67db/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d7183f126a97fedb177bb2869ecb7eeebb9b8194d8d3e2c2075bb5d11fe866f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/408faab9-57a4-4c18-86ae-68772011de21/volumes/kubernetes.io~projected/kube-api-access-mdphh DeviceMajor:0 DeviceMinor:221 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:225 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc5110ed-77a9-443b-b64f-5309fadadd78/volumes/kubernetes.io~projected/kube-api-access-9zfbv DeviceMajor:0 DeviceMinor:345 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d4dcfbc57887f3ae26e161794606c8a4eec7b2c4a0a5c246c7b3425f645a1fa/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92169b77fc7554447d2dd9fb89c3b6c9fb184a3190082a40f1f251f1d1e0588f/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc5110ed-77a9-443b-b64f-5309fadadd78/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:344 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c83cbbe5-3d68-4067-a0e9-c38f0f72df38/volumes/kubernetes.io~projected/kube-api-access-wjz6q DeviceMajor:0 DeviceMinor:352 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99cd239f879c4d8945493c1c5fd1bbbc9d46e942c27f545275e820265d06f214/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e209316-e440-4bdf-b2d9-cc79ac3e670f/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:213 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c44bba77-c0b2-49cc-92c1-836cf8013950/volumes/kubernetes.io~projected/kube-api-access-44dtj DeviceMajor:0 DeviceMinor:204 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24f3f5a6d858460eadc4193742eacb162f14f47f097c451919ae080b66f127bd/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c416b494-9d26-466e-b840-103ce32abf45/volumes/kubernetes.io~projected/kube-api-access-j45q9 DeviceMajor:0 DeviceMinor:339 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c8302492-d696-40ad-bffd-431b35319e22/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f53b8d98-bbb7-4108-a65d-6a15ab782bee/volumes/kubernetes.io~projected/kube-api-access-dvtlp DeviceMajor:0 DeviceMinor:205 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5266b841-4495-40ff-8277-a8c7b6fdba80/volumes/kubernetes.io~projected/kube-api-access-n42j6 DeviceMajor:0 DeviceMinor:220 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2e209316-e440-4bdf-b2d9-cc79ac3e670f/volumes/kubernetes.io~projected/kube-api-access-tq9rt DeviceMajor:0 DeviceMinor:222 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34748730-f975-4008-9a37-dad912c7e930/volumes/kubernetes.io~projected/kube-api-access-7t88q DeviceMajor:0 DeviceMinor:219 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:226 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f59717aa-c796-4074-b63f-2fc7e13f3633/volumes/kubernetes.io~projected/kube-api-access-5rbld DeviceMajor:0 DeviceMinor:239 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f53b8d98-bbb7-4108-a65d-6a15ab782bee/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/db65a944-e395-4d70-9807-c061aecf0584/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:227 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed/volumes/kubernetes.io~projected/kube-api-access-z89wv DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2236639900fd66a76022f581d57a34b0c3a890430f3ef984b48b96a5cddd6896/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a12afa6f7ebfd8eeaba6e68fc9759bfb06eb34678d2cd2af4a1a9c44a43fe539/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4b98ab84513804906c41697e62ab0a3f72ac8f3870f11868c9f05645281992b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4479da8f52de7c7e460cf7dd7e12df5a099ceb3a815f3d483c29d4c3781924d5/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2236639900fd66a MacAddress:3e:75:8b:f3:10:60 Speed:10000 Mtu:8901} {Name:24f3f5a6d858460 MacAddress:62:d0:0e:f4:71:fe Speed:10000 Mtu:8901} {Name:337dfcfdc8d1c82 MacAddress:2a:20:25:9f:40:fd Speed:10000 Mtu:8901} {Name:4479da8f52de7c7 MacAddress:9a:09:62:a2:1a:5d Speed:10000 Mtu:8901} {Name:4f91ec2ee852545 MacAddress:1e:6a:09:ec:b6:1d Speed:10000 Mtu:8901} {Name:5037dff5fe58111 MacAddress:ae:05:64:02:ed:1b Speed:10000 Mtu:8901} {Name:5ce410fcc02e436 MacAddress:e6:60:4b:43:47:8a Speed:10000 Mtu:8901} {Name:92169b77fc75544 MacAddress:d6:9a:63:e6:c8:90 Speed:10000 Mtu:8901} {Name:99cd239f879c4d8 MacAddress:4a:bc:47:24:83:a6 Speed:10000 Mtu:8901} {Name:a12afa6f7ebfd8e MacAddress:aa:be:b7:fe:94:96 Speed:10000 Mtu:8901} {Name:a7aea42d2a49816 MacAddress:72:a7:25:c5:5b:47 Speed:10000 Mtu:8901} {Name:ad0ab66cdd8de4c MacAddress:2a:26:c2:bb:69:42 Speed:10000 Mtu:8901} {Name:b228b0c3a72c882 MacAddress:46:e6:9c:2b:0e:4a Speed:10000 Mtu:8901} {Name:be45bfd55cd4bd3 MacAddress:8e:78:d9:9b:20:ba Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:cc:10:c9:9f Speed:0 Mtu:9001} {Name:br-int MacAddress:56:a7:aa:3b:55:2e Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:cc:10:c9:9f Speed:0 Mtu:9001} {Name:ff9b9b39519a0df MacAddress:a2:4c:f6:7c:71:23 Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d6:25:92:a3:5c:77 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.880695    4935 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.880745    4935 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.881808    4935 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.881840    4935 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-125.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.882186    4935 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.882199    4935 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.882218    4935 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.882244    4935 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.882271    4935 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.882929    4935 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.883128    4935 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.883180    4935 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.883199    4935 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.883215    4935 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.883229    4935 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.883241    4935 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.884245    4935 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.884259    4935 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:28:47.887397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.884868    4935 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:28:47.896070 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896029    4935 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:28:47.896070 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896070    4935 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:28:47.896205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896195    4935 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 00:28:47.896413 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896402    4935 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:28:47.896442 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896431    4935 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:28:47.896544 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896535    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:28:47.896573 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896550    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:28:47.896573 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896556    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:28:47.896573 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896562    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:28:47.896573 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896568    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:28:47.896573 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896573    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:28:47.896697 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896580    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:28:47.896697 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896589    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:28:47.896697 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896596    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:28:47.896697 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896601    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:28:47.896697 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896610    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:28:47.896697 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896619    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:28:47.896697 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896644    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:28:47.896697 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.896652    4935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:28:47.900299 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.900285    4935 server.go:1263] "Started kubelet"
Oct 31 00:28:47.900989 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.900401    4935 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:28:47.900989 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.900478    4935 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:28:47.900989 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.900564    4935 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:28:47.900989 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.900911    4935 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:28:47.900977 ip-10-0-132-125 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:28:47.902569 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.902057    4935 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:28:47.909084 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.908504    4935 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:28:47.909084 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.908516    4935 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:28:47.909084 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.908627    4935 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:23:08 +0000 UTC" deadline="2028-05-15 05:48:02.045111216 +0000 UTC"
Oct 31 00:28:47.909084 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.908651    4935 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13493h19m14.136463109s"
Oct 31 00:28:47.909084 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.908678    4935 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:28:47.909084 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.908690    4935 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:28:47.909084 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.908758    4935 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:28:47.910624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.909184    4935 factory.go:55] Registering systemd factory
Oct 31 00:28:47.910624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.909266    4935 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:28:47.910624 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:47.909233    4935 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:47.910624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.909629    4935 factory.go:153] Registering CRI-O factory
Oct 31 00:28:47.910624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.909644    4935 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:28:47.910624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.909704    4935 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:28:47.910624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.909731    4935 factory.go:103] Registering Raw factory
Oct 31 00:28:47.910624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.909750    4935 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:28:47.910997 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.910654    4935 manager.go:319] Starting recovery of all containers
Oct 31 00:28:47.911766 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911727    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8302492-d696-40ad-bffd-431b35319e22" volumeName="kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-tmp" seLinuxMountContext=""
Oct 31 00:28:47.911865 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911770    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a996538-9d06-447f-bf87-e7dbb8f078c6" volumeName="kubernetes.io/configmap/2a996538-9d06-447f-bf87-e7dbb8f078c6-config-volume" seLinuxMountContext=""
Oct 31 00:28:47.911865 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911789    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e209316-e440-4bdf-b2d9-cc79ac3e670f" volumeName="kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-stats-auth" seLinuxMountContext=""
Oct 31 00:28:47.911865 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911802    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ecc909-648d-4848-9f49-464285458129" volumeName="kubernetes.io/projected/54ecc909-648d-4848-9f49-464285458129-kube-api-access-2d5cn" seLinuxMountContext=""
Oct 31 00:28:47.911865 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911815    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76cd3516-aff9-4de2-a063-c30407f68564" volumeName="kubernetes.io/empty-dir/76cd3516-aff9-4de2-a063-c30407f68564-ca-trust-extracted" seLinuxMountContext=""
Oct 31 00:28:47.911865 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911830    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81be0f35-6496-46d5-a27b-46bd185e3993" volumeName="kubernetes.io/projected/81be0f35-6496-46d5-a27b-46bd185e3993-kube-api-access-jb2gk" seLinuxMountContext=""
Oct 31 00:28:47.911865 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911842    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a27a3030-72b2-4223-b95e-c3997a8eb0f6" volumeName="kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-config" seLinuxMountContext=""
Oct 31 00:28:47.911865 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911854    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1402f00-5adc-44cb-961c-a2556e51529b" volumeName="kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911871    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76cd3516-aff9-4de2-a063-c30407f68564" volumeName="kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-registry-certificates" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911885    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76cd3516-aff9-4de2-a063-c30407f68564" volumeName="kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-installation-pull-secrets" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911898    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a27a3030-72b2-4223-b95e-c3997a8eb0f6" volumeName="kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911909    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44bba77-c0b2-49cc-92c1-836cf8013950" volumeName="kubernetes.io/secret/c44bba77-c0b2-49cc-92c1-836cf8013950-serving-cert" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911921    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c83cbbe5-3d68-4067-a0e9-c38f0f72df38" volumeName="kubernetes.io/projected/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-api-access-wjz6q" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911936    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6511745-7315-463c-bc25-0e3e1fd6bf42" volumeName="kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911950    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7ef07d0-5483-4176-bfb2-4e03c9be4498" volumeName="kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-daemon-config" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911962    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed" volumeName="kubernetes.io/projected/dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed-kube-api-access-z89wv" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911973    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6511745-7315-463c-bc25-0e3e1fd6bf42" volumeName="kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-certificates" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.911989    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5266b841-4495-40ff-8277-a8c7b6fdba80" volumeName="kubernetes.io/projected/5266b841-4495-40ff-8277-a8c7b6fdba80-kube-api-access-n42j6" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912003    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a27a3030-72b2-4223-b95e-c3997a8eb0f6" volumeName="kubernetes.io/secret/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912015    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1402f00-5adc-44cb-961c-a2556e51529b" volumeName="kubernetes.io/projected/b1402f00-5adc-44cb-961c-a2556e51529b-kube-api-access-6lvrb" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912028    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53b8d98-bbb7-4108-a65d-6a15ab782bee" volumeName="kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912057    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="261b0474-eb3d-4f3c-b33d-01856306e928" volumeName="kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912069    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76cd3516-aff9-4de2-a063-c30407f68564" volumeName="kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-trusted-ca" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912082    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="990975fd-2466-4224-acbe-6269e49a9ef9" volumeName="kubernetes.io/configmap/990975fd-2466-4224-acbe-6269e49a9ef9-config" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912095    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="990975fd-2466-4224-acbe-6269e49a9ef9" volumeName="kubernetes.io/secret/990975fd-2466-4224-acbe-6269e49a9ef9-serving-cert" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912108    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a27a3030-72b2-4223-b95e-c3997a8eb0f6" volumeName="kubernetes.io/projected/a27a3030-72b2-4223-b95e-c3997a8eb0f6-kube-api-access-86wrt" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912120    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44bba77-c0b2-49cc-92c1-836cf8013950" volumeName="kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-config" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912148    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44bba77-c0b2-49cc-92c1-836cf8013950" volumeName="kubernetes.io/projected/c44bba77-c0b2-49cc-92c1-836cf8013950-kube-api-access-44dtj" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912169    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc5110ed-77a9-443b-b64f-5309fadadd78" volumeName="kubernetes.io/projected/dc5110ed-77a9-443b-b64f-5309fadadd78-kube-api-access-9zfbv" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912187    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="408faab9-57a4-4c18-86ae-68772011de21" volumeName="kubernetes.io/projected/408faab9-57a4-4c18-86ae-68772011de21-kube-api-access-mdphh" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912205    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ab38846-5d17-4bf1-92c2-ac823690fc98" volumeName="kubernetes.io/configmap/7ab38846-5d17-4bf1-92c2-ac823690fc98-konnectivity-ca" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912221    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5703958-5751-4c4d-b979-01585c84cd46" volumeName="kubernetes.io/configmap/f5703958-5751-4c4d-b979-01585c84cd46-serviceca" seLinuxMountContext=""
Oct 31 00:28:47.912222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912239    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="693a3c1f-8228-4691-bd99-f05fdf784cd4" volumeName="kubernetes.io/projected/693a3c1f-8228-4691-bd99-f05fdf784cd4-kube-api-access-5tz7l" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912257    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="967855d0-8720-446c-a8df-9e2b427533b3" volumeName="kubernetes.io/projected/967855d0-8720-446c-a8df-9e2b427533b3-kube-api-access-czwhh" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912272    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1402f00-5adc-44cb-961c-a2556e51529b" volumeName="kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912322    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6511745-7315-463c-bc25-0e3e1fd6bf42" volumeName="kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-bound-sa-token" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912340    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="990975fd-2466-4224-acbe-6269e49a9ef9" volumeName="kubernetes.io/projected/990975fd-2466-4224-acbe-6269e49a9ef9-kube-api-access-cbmnh" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912356    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c44bba77-c0b2-49cc-92c1-836cf8013950" volumeName="kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-trusted-ca" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912374    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6511745-7315-463c-bc25-0e3e1fd6bf42" volumeName="kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-installation-pull-secrets" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912947    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67064c-ba5d-4780-9257-9df20f304f29" volumeName="kubernetes.io/projected/0e67064c-ba5d-4780-9257-9df20f304f29-kube-api-access-r4f8k" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912958    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a996538-9d06-447f-bf87-e7dbb8f078c6" volumeName="kubernetes.io/projected/2a996538-9d06-447f-bf87-e7dbb8f078c6-kube-api-access-d7lmn" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912970    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53b8d98-bbb7-4108-a65d-6a15ab782bee" volumeName="kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-service-ca-bundle" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912982    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8302492-d696-40ad-bffd-431b35319e22" volumeName="kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-etc-tuned" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.912995    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9fad5d0-0d3a-4ebe-a07b-1899400059a7" volumeName="kubernetes.io/configmap/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-iptables-alerter-script" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913007    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc77171d-558b-40c2-aa20-8e45c1693030" volumeName="kubernetes.io/projected/bc77171d-558b-40c2-aa20-8e45c1693030-kube-api-access-b79cl" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913019    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34d3cbf5-134e-4f86-8b33-04ef38cdb685" volumeName="kubernetes.io/projected/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kube-api-access-2q9pr" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913052    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76cd3516-aff9-4de2-a063-c30407f68564" volumeName="kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-kube-api-access-xfmz7" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913065    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ab38846-5d17-4bf1-92c2-ac823690fc98" volumeName="kubernetes.io/secret/7ab38846-5d17-4bf1-92c2-ac823690fc98-agent-certs" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913097    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9fad5d0-0d3a-4ebe-a07b-1899400059a7" volumeName="kubernetes.io/projected/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-kube-api-access-495pb" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913110    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c416b494-9d26-466e-b840-103ce32abf45" volumeName="kubernetes.io/projected/c416b494-9d26-466e-b840-103ce32abf45-kube-api-access-j45q9" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913122    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53b8d98-bbb7-4108-a65d-6a15ab782bee" volumeName="kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-snapshots" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913134    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5703958-5751-4c4d-b979-01585c84cd46" volumeName="kubernetes.io/projected/f5703958-5751-4c4d-b979-01585c84cd46-kube-api-access-dnvlx" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913147    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48c64d8-b46d-48e0-830e-d9ec6a92ff06" volumeName="kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913158    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34748730-f975-4008-9a37-dad912c7e930" volumeName="kubernetes.io/projected/34748730-f975-4008-9a37-dad912c7e930-kube-api-access-7t88q" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913169    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a27a3030-72b2-4223-b95e-c3997a8eb0f6" volumeName="kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-env-overrides" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913181    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc5110ed-77a9-443b-b64f-5309fadadd78" volumeName="kubernetes.io/secret/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-key" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913203    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f59717aa-c796-4074-b63f-2fc7e13f3633" volumeName="kubernetes.io/secret/f59717aa-c796-4074-b63f-2fc7e13f3633-serving-cert" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913216    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8302492-d696-40ad-bffd-431b35319e22" volumeName="kubernetes.io/projected/c8302492-d696-40ad-bffd-431b35319e22-kube-api-access-44wtx" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913226    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="693a3c1f-8228-4691-bd99-f05fdf784cd4" volumeName="kubernetes.io/empty-dir/693a3c1f-8228-4691-bd99-f05fdf784cd4-tmp-dir" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913238    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f59717aa-c796-4074-b63f-2fc7e13f3633" volumeName="kubernetes.io/projected/f59717aa-c796-4074-b63f-2fc7e13f3633-kube-api-access-5rbld" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913249    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7ef07d0-5483-4176-bfb2-4e03c9be4498" volumeName="kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913260    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53b8d98-bbb7-4108-a65d-6a15ab782bee" volumeName="kubernetes.io/secret/f53b8d98-bbb7-4108-a65d-6a15ab782bee-serving-cert" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913273    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a996538-9d06-447f-bf87-e7dbb8f078c6" volumeName="kubernetes.io/empty-dir/2a996538-9d06-447f-bf87-e7dbb8f078c6-tmp-dir" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913285    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e209316-e440-4bdf-b2d9-cc79ac3e670f" volumeName="kubernetes.io/projected/2e209316-e440-4bdf-b2d9-cc79ac3e670f-kube-api-access-tq9rt" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913296    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e209316-e440-4bdf-b2d9-cc79ac3e670f" volumeName="kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-default-certificate" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913317    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5266b841-4495-40ff-8277-a8c7b6fdba80" volumeName="kubernetes.io/configmap/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-config" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913331    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76cd3516-aff9-4de2-a063-c30407f68564" volumeName="kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913341    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6511745-7315-463c-bc25-0e3e1fd6bf42" volumeName="kubernetes.io/empty-dir/d6511745-7315-463c-bc25-0e3e1fd6bf42-ca-trust-extracted" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913354    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db65a944-e395-4d70-9807-c061aecf0584" volumeName="kubernetes.io/secret/db65a944-e395-4d70-9807-c061aecf0584-cert" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913365    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53b8d98-bbb7-4108-a65d-6a15ab782bee" volumeName="kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-tmp" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913377    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ecc909-648d-4848-9f49-464285458129" volumeName="kubernetes.io/configmap/54ecc909-648d-4848-9f49-464285458129-telemetry-config" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913390    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76cd3516-aff9-4de2-a063-c30407f68564" volumeName="kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-bound-sa-token" seLinuxMountContext=""
Oct 31 00:28:47.913372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913401    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c83cbbe5-3d68-4067-a0e9-c38f0f72df38" volumeName="kubernetes.io/empty-dir/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-data-volume" seLinuxMountContext=""
Oct 31 00:28:47.916329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913411    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d543da15-0c67-4912-a39e-04235a6afa20" volumeName="kubernetes.io/configmap/d543da15-0c67-4912-a39e-04235a6afa20-nginx-conf" seLinuxMountContext=""
Oct 31 00:28:47.916329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913424    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6511745-7315-463c-bc25-0e3e1fd6bf42" volumeName="kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-trusted-ca" seLinuxMountContext=""
Oct 31 00:28:47.916329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913434    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f53b8d98-bbb7-4108-a65d-6a15ab782bee" volumeName="kubernetes.io/projected/f53b8d98-bbb7-4108-a65d-6a15ab782bee-kube-api-access-dvtlp" seLinuxMountContext=""
Oct 31 00:28:47.916329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913446    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f59717aa-c796-4074-b63f-2fc7e13f3633" volumeName="kubernetes.io/configmap/f59717aa-c796-4074-b63f-2fc7e13f3633-config" seLinuxMountContext=""
Oct 31 00:28:47.916329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913457    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1402f00-5adc-44cb-961c-a2556e51529b" volumeName="kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:28:47.916329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913468    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6511745-7315-463c-bc25-0e3e1fd6bf42" volumeName="kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-kube-api-access-9ppwn" seLinuxMountContext=""
Oct 31 00:28:47.916329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913479    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7ef07d0-5483-4176-bfb2-4e03c9be4498" volumeName="kubernetes.io/projected/d7ef07d0-5483-4176-bfb2-4e03c9be4498-kube-api-access-kfs9t" seLinuxMountContext=""
Oct 31 00:28:47.916329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913489    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db65a944-e395-4d70-9807-c061aecf0584" volumeName="kubernetes.io/projected/db65a944-e395-4d70-9807-c061aecf0584-kube-api-access-rpz4f" seLinuxMountContext=""
Oct 31 00:28:47.916329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913500    4935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc5110ed-77a9-443b-b64f-5309fadadd78" volumeName="kubernetes.io/configmap/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-cabundle" seLinuxMountContext=""
Oct 31 00:28:47.916329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913511    4935 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:28:47.916329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.913518    4935 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:28:47.916973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.916952    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/1.log"
Oct 31 00:28:47.919762 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.919737    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/1.log"
Oct 31 00:28:47.935217 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.935181    4935 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:28:47.936628 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.936598    4935 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:28:47.936628 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.936631    4935 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:28:47.936758 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.936659    4935 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:28:47.936758 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:47.936705    4935 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 00:28:47.959927 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.959900    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/1.log"
Oct 31 00:28:47.960097 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.959932    4935 generic.go:356] "Generic (PLEG): container finished" podID="c44bba77-c0b2-49cc-92c1-836cf8013950" containerID="1010115f821e74be8029255505b4165efa642f8bce3ec324faae93519f703f55" exitCode=255
Oct 31 00:28:47.969955 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.969915    4935 generic.go:356] "Generic (PLEG): container finished" podID="4919af72af1b023905c8a7373620fc2c" containerID="03a9b1d044b12443a24cd27838bf3462545f1a6cd800867db806a94b2a83b5bb" exitCode=0
Oct 31 00:28:47.993979 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.993938    4935 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="fa6f1d14d81dc2c1b28b9a8c1845f34ef789d30ee19b96bfcd6d37d82b46270b" exitCode=0
Oct 31 00:28:47.993979 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.993972    4935 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="cf7c7e86295c6d7d80f2c411f27774e2fc3121dac236ccb8a55ed12c640048f7" exitCode=0
Oct 31 00:28:47.993979 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.993983    4935 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="a10696b588f51c2de21f024adcd69c0af4f1227ceaae7de50f22cd3538b0a50d" exitCode=0
Oct 31 00:28:47.994271 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.993992    4935 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="509c859dc7abbaae7d5ed8dc71f1944b1de2064b1bab095f1322199b8be982b7" exitCode=0
Oct 31 00:28:47.994271 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.994003    4935 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="5609e590a06a63efcf736e76edfab193b90cf8449d45a89991f604131f6cb01a" exitCode=0
Oct 31 00:28:47.994271 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:47.994011    4935 generic.go:356] "Generic (PLEG): container finished" podID="b1402f00-5adc-44cb-961c-a2556e51529b" containerID="a43b1a197a33502d9485f0d793c3ba4d9fc5d2e6a45ae2f7e63a09ef14e5fe2f" exitCode=0
Oct 31 00:28:48.009109 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.007016    4935 manager.go:324] Recovery completed
Oct 31 00:28:48.009540 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:48.009517    4935 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:48.026934 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.026909    4935 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:48.028458 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.028436    4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:48.028653 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.028471    4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:48.028653 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.028487    4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:48.031378 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031358    4935 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:28:48.031378 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031376    4935 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:28:48.031524 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031384    4935 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:48.031524 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031398    4935 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:28:48.031655 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031550    4935 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 00:28:48.031655 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031565    4935 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 00:28:48.031655 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031592    4935 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 00:28:48.031655 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031600    4935 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 00:28:48.031655 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031607    4935 policy_none.go:50] "Start"
Oct 31 00:28:48.031655 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031619    4935 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:28:48.031655 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031633    4935 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:28:48.031954 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031810    4935 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 00:28:48.031954 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031821    4935 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 00:28:48.031954 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.031829    4935 policy_none.go:44] "Start"
Oct 31 00:28:48.032394 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.032373    4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:48.032507 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.032404    4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:48.032507 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.032414    4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:48.036724 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.036692    4935 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:28:48.036832 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:48.036758    4935 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 00:28:48.042318 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.042301    4935 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:28:48.042419 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:48.042333    4935 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:28:48.042419 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.042345    4935 server.go:87] "Starting device plugin registration server"
Oct 31 00:28:48.042664 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.042650    4935 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:28:48.042737 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.042664    4935 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:28:48.042787 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.042760    4935 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:28:48.042886 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.042856    4935 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:28:48.042936 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.042893    4935 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:28:48.042936 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.042900    4935 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:28:48.043710 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.042984    4935 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:28:48.042883931Z","UUID":"8c602d29-34a5-41a8-b199-3ca8aafd1374","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:28:48.044864 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.044829    4935 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:28:48.044864 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.044862    4935 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:28:48.045017 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:48.044988    4935 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:28:48.045401 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:48.045353    4935 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-125.us-west-2.compute.internal\" not found"
Oct 31 00:28:48.085719 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.085689    4935 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:48.085719 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.085707    4935 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:48.085941 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.085825    4935 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 00:28:48.087491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.087478    4935 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:48.134048 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.134007    4935 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:28:48.143393 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.143366    4935 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:48.144460 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.144433    4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:48.144556 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.144474    4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:48.144556 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.144489    4935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-125.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:48.144556 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.144529    4935 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.237346 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.237299    4935 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"]
Oct 31 00:28:48.237939 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.237906    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal" event={"ID":"859c761ccba0302346f6d0c201b0e803","Type":"ContainerStarted","Data":"de4379c3480812a1238e2198b0f2eeb00b8faec8d5a2e95f6cceb1e7297b1f22"}
Oct 31 00:28:48.237939 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.237938    4935 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.238101 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.237950    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal" event={"ID":"859c761ccba0302346f6d0c201b0e803","Type":"ContainerStarted","Data":"c4b98ab84513804906c41697e62ab0a3f72ac8f3870f11868c9f05645281992b"}
Oct 31 00:28:48.238101 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.237968    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal" event={"ID":"4919af72af1b023905c8a7373620fc2c","Type":"ContainerStarted","Data":"a22cba39c74c7ad19f981d728ea0d46de7f88b00ca1034851d17f8bb7f781c5b"}
Oct 31 00:28:48.238101 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.237980    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal" event={"ID":"4919af72af1b023905c8a7373620fc2c","Type":"ContainerDied","Data":"03a9b1d044b12443a24cd27838bf3462545f1a6cd800867db806a94b2a83b5bb"}
Oct 31 00:28:48.238101 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.237992    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal" event={"ID":"4919af72af1b023905c8a7373620fc2c","Type":"ContainerStarted","Data":"3d4dcfbc57887f3ae26e161794606c8a4eec7b2c4a0a5c246c7b3425f645a1fa"}
Oct 31 00:28:48.238216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.238146    4935 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.250857 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.250837    4935 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.250919 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.250901    4935 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.296818 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.296736    4935 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:48.296818 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:48.296788    4935 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.298195 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.298179    4935 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:48.298274 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:48.298217    4935 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.388007 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.387966    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/859c761ccba0302346f6d0c201b0e803-config\") pod \"kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"859c761ccba0302346f6d0c201b0e803\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.388007 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.388006    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.388249 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.388104    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.488495 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.488459    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.488669 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.488506    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.488669 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.488528    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/859c761ccba0302346f6d0c201b0e803-config\") pod \"kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"859c761ccba0302346f6d0c201b0e803\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.488669 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.488474    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.488669 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.488566    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/859c761ccba0302346f6d0c201b0e803-config\") pod \"kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"859c761ccba0302346f6d0c201b0e803\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.488669 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.488597    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4919af72af1b023905c8a7373620fc2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" (UID: \"4919af72af1b023905c8a7373620fc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:48.884417 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:48.884314    4935 apiserver.go:52] "Watching apiserver"
Oct 31 00:28:49.001006 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.000976    4935 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:49.001171 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.001151    4935 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:49.030263 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.030233    4935 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:28:49.032387 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.032344    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-grc8r","openshift-multus/multus-additional-cni-plugins-h8q7j","openshift-service-ca/service-ca-5fffc97bdf-cn9rf","openshift-image-registry/node-ca-fvksv","openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp","openshift-insights/insights-runtime-extractor-x7l7z","openshift-ingress/router-default-55c864699c-n4cfn","openshift-network-console/networking-console-plugin-86c478bb88-9lmll","kube-system/kas-connection-checker-685895c7c9-tkpbr","openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr","kube-system/kas-connection-checker-685895c7c9-hvfpm","openshift-dns/node-resolver-29ctl","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt","openshift-image-registry/image-registry-7f66ccc5cb-h7cvd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s","openshift-insights/insights-operator-7c4986d449-4sch4","openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8","openshift-multus/network-metrics-daemon-7smc9","openshift-network-operator/iptables-alerter-8p6vl","openshift-image-registry/image-registry-6cfc76855c-xm22c","openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn","kube-system/konnectivity-agent-l64l2","openshift-multus/multus-76vsc","openshift-network-diagnostics/network-check-target-pdpbv","kube-system/global-pull-secret-syncer-wdzhr","kube-system/kas-connection-checker-685895c7c9-8tt4s","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk","openshift-cluster-node-tuning-operator/tuned-sxssn","openshift-ingress-canary/ingress-canary-jmdnc","openshift-ovn-kubernetes/ovnkube-node-z98bg","kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal","openshift-console-operator/console-operator-7cd56b8dbc-pcbjl","openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2"]
Oct 31 00:28:49.036375 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.036351    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.037231 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.037210    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.037814 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.037796    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:49.038502 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.038483    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:49.040244 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.040224    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:49.046747 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.046726    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:49.050941 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.050877    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.052714 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.052695    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:49.057016 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.056995    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.059622 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.059594    4935 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:49.059695 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.059647    4935 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:49.060931 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.060911    4935 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:28:49.061030 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.060959    4935 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 00:28:49.085377 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.085351    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 00:28:49.085545 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.085530    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qjlzt\""
Oct 31 00:28:49.087951 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.087931    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.088104 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.088089    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.088173 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.088093    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.088173 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.088095    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.088173 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.088122    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 00:28:49.088335 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.088179    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.113947 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.113916    4935 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:28:49.134129 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.134095    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 00:28:49.135064 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.134968    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 00:28:49.135064 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135003    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.135251 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135096    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 00:28:49.135251 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135106    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.135251 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135166    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:28:49.135251 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135219    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.135832 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135813    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 00:28:49.135956 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135796    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 00:28:49.135956 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135849    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bz5sd\""
Oct 31 00:28:49.135956 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135886    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 00:28:49.135956 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135908    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vk6nb\""
Oct 31 00:28:49.135956 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135915    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bjgrg\""
Oct 31 00:28:49.135956 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135918    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135973    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.135986    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136017    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136074    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pkx6z\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136127    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136141    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136207    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136214    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136224    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136263    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136273    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.136282 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136274    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 00:28:49.136790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136294    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-nx2tp\""
Oct 31 00:28:49.136790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136344    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.136790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136368    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 00:28:49.136790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136404    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.136790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136412    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.136790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136447    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.136790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136561    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.136790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136631    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.136790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136664    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:28:49.136790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136695    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136806    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136812    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136859    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136877    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136886    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136888    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136905    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136915    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136924    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136932    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136948    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136949    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136961    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136970    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136963    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136981    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.136999    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137029    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137056    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137072    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 00:28:49.137205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137125    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 00:28:49.137816 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137320    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:28:49.137816 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137362    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tsjd8\""
Oct 31 00:28:49.137816 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137376    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:28:49.137816 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137523    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s5c24\""
Oct 31 00:28:49.137816 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137571    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.137816 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137624    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-n47jx\""
Oct 31 00:28:49.137816 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137657    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.137816 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137686    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.137816 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137718    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-klgwd\""
Oct 31 00:28:49.137816 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137754    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-z9m2w\""
Oct 31 00:28:49.137816 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137784    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.138255 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137849    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:28:49.138255 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137868    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bv698\""
Oct 31 00:28:49.138255 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137929    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.138255 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137937    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l8m77\""
Oct 31 00:28:49.138255 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.137984    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-prxzz\""
Oct 31 00:28:49.138255 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138072    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzctz\""
Oct 31 00:28:49.138255 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138115    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2sfqr\""
Oct 31 00:28:49.138508 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138259    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:28:49.138508 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138321    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.138508 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138435    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.138508 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138478    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z6hfz\""
Oct 31 00:28:49.138630 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138546    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4l8km\""
Oct 31 00:28:49.138630 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138571    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 00:28:49.138630 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138607    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6z87r\""
Oct 31 00:28:49.138722 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138659    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:28:49.138722 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138710    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5b488\""
Oct 31 00:28:49.138831 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138770    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t6xpx\""
Oct 31 00:28:49.138831 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138821    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bpzsc\""
Oct 31 00:28:49.138976 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138873    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5vb9k\""
Oct 31 00:28:49.138976 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138901    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zm47m\""
Oct 31 00:28:49.138976 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.138949    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wzhm5\""
Oct 31 00:28:49.139107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.139022    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9b4md\""
Oct 31 00:28:49.139107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.139105    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-g8859\""
Oct 31 00:28:49.139199 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.139188    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:28:49.141839 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.141806    4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:49.147446 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147399    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:28:49.147582 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147558    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.147582 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147564    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.147770 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147642    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 00:28:49.147770 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147683    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.147770 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147687    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:28:49.147770 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147712    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:28:49.147927 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147819    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.147927 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147854    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 00:28:49.147927 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147859    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:28:49.147927 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147858    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:28:49.147927 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147905    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:28:49.147927 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.147906    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:28:49.194271 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194235    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.194271 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194277    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-ovn\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.194586 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194311    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-trusted-ca\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.194586 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194375    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-system-cni-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.194586 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194418    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-cnibin\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.194586 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194454    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ppwn\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-kube-api-access-9ppwn\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.194586 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194478    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.194883 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194728    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-trusted-ca\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.194883 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194780    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-service-ca-bundle\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.194883 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194805    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f53b8d98-bbb7-4108-a65d-6a15ab782bee-serving-cert\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.194883 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194845    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysconfig\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.194883 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194873    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cnibin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.195186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194901    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-netns\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.195186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194929    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/54ecc909-648d-4848-9f49-464285458129-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:49.195186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194955    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.195186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.194985    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f53b8d98-bbb7-4108-a65d-6a15ab782bee-serving-cert\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.195186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195007    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c44bba77-c0b2-49cc-92c1-836cf8013950-serving-cert\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:49.195186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195086    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-dbus\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:49.195186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195115    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rbld\" (UniqueName: \"kubernetes.io/projected/f59717aa-c796-4074-b63f-2fc7e13f3633-kube-api-access-5rbld\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:49.195186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195140    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.195186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195187    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-conf-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195211    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfs9t\" (UniqueName: \"kubernetes.io/projected/d7ef07d0-5483-4176-bfb2-4e03c9be4498-kube-api-access-kfs9t\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195247    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195273    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5cn\" (UniqueName: \"kubernetes.io/projected/54ecc909-648d-4848-9f49-464285458129-kube-api-access-2d5cn\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195297    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-registration-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195324    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9pr\" (UniqueName: \"kubernetes.io/projected/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kube-api-access-2q9pr\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195427    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-iptables-alerter-script\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195454    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-k8s-cni-cncf-io\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195496    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-certificates\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195522    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195558    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n42j6\" (UniqueName: \"kubernetes.io/projected/5266b841-4495-40ff-8277-a8c7b6fdba80-kube-api-access-n42j6\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195582    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-cabundle\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195604    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-key\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195625    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-service-ca-bundle\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195628    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45q9\" (UniqueName: \"kubernetes.io/projected/c416b494-9d26-466e-b840-103ce32abf45-kube-api-access-j45q9\") pod \"migrator-66df5d496b-5wmv2\" (UID: \"c416b494-9d26-466e-b840-103ce32abf45\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195667    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-installation-pull-secrets\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195696    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/990975fd-2466-4224-acbe-6269e49a9ef9-serving-cert\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:49.195738 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195725    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-systemd-units\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195789    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-bin\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195815    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-trusted-ca\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195839    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-bound-sa-token\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195864    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-kubelet-config\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195887    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5703958-5751-4c4d-b979-01585c84cd46-host\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195911    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6511745-7315-463c-bc25-0e3e1fd6bf42-ca-trust-extracted\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195936    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-sys\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195958    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-host\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.195984    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-etc-tuned\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196009    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ab38846-5d17-4bf1-92c2-ac823690fc98-agent-certs\") pod \"konnectivity-agent-l64l2\" (UID: \"7ab38846-5d17-4bf1-92c2-ac823690fc98\") " pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196062    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-var-lib-kubelet\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196198    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ab38846-5d17-4bf1-92c2-ac823690fc98-konnectivity-ca\") pod \"konnectivity-agent-l64l2\" (UID: \"7ab38846-5d17-4bf1-92c2-ac823690fc98\") " pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196224    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-etc-kubernetes\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196251    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t88q\" (UniqueName: \"kubernetes.io/projected/34748730-f975-4008-9a37-dad912c7e930-kube-api-access-7t88q\") pod \"volume-data-source-validator-58c9f5f46d-qjwgt\" (UID: \"34748730-f975-4008-9a37-dad912c7e930\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196278    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-crio-socket\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196301    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-slash\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196325    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196351    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-env-overrides\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196376    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86wrt\" (UniqueName: \"kubernetes.io/projected/a27a3030-72b2-4223-b95e-c3997a8eb0f6-kube-api-access-86wrt\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196401    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196426    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czwhh\" (UniqueName: \"kubernetes.io/projected/967855d0-8720-446c-a8df-9e2b427533b3-kube-api-access-czwhh\") pod \"network-check-source-6f78d59f99-5dlfp\" (UID: \"967855d0-8720-446c-a8df-9e2b427533b3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196452    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpz4f\" (UniqueName: \"kubernetes.io/projected/db65a944-e395-4d70-9807-c061aecf0584-kube-api-access-rpz4f\") pod \"ingress-canary-jmdnc\" (UID: \"db65a944-e395-4d70-9807-c061aecf0584\") " pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196478    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a996538-9d06-447f-bf87-e7dbb8f078c6-tmp-dir\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196503    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f59717aa-c796-4074-b63f-2fc7e13f3633-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196528    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196553    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-sys-fs\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196198    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/54ecc909-648d-4848-9f49-464285458129-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196716    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-tmp\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.196739 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196743    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-config\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196768    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-trusted-ca\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196793    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-host-slash\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196819    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfmz7\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-kube-api-access-xfmz7\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196853    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfbv\" (UniqueName: \"kubernetes.io/projected/dc5110ed-77a9-443b-b64f-5309fadadd78-kube-api-access-9zfbv\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196878    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9rt\" (UniqueName: \"kubernetes.io/projected/2e209316-e440-4bdf-b2d9-cc79ac3e670f-kube-api-access-tq9rt\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196905    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693a3c1f-8228-4691-bd99-f05fdf784cd4-tmp-dir\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196930    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-kubelet\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196955    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-daemon-config\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.196980    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-netd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.197006    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.197149    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-etc-tuned\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.197423    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ab38846-5d17-4bf1-92c2-ac823690fc98-agent-certs\") pod \"konnectivity-agent-l64l2\" (UID: \"7ab38846-5d17-4bf1-92c2-ac823690fc98\") " pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.197633    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ab38846-5d17-4bf1-92c2-ac823690fc98-konnectivity-ca\") pod \"konnectivity-agent-l64l2\" (UID: \"7ab38846-5d17-4bf1-92c2-ac823690fc98\") " pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.197709    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-trusted-ca\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.197982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.197971    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-installation-pull-secrets\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.198118    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-env-overrides\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.198147    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/990975fd-2466-4224-acbe-6269e49a9ef9-serving-cert\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.198400    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c44bba77-c0b2-49cc-92c1-836cf8013950-serving-cert\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.198448    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6511745-7315-463c-bc25-0e3e1fd6bf42-ca-trust-extracted\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.198466    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.198640    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-tmp\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.198838    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-iptables-alerter-script\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.198973    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-certificates\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.198974    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-trusted-ca\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.198996    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-cabundle\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199265    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c44bba77-c0b2-49cc-92c1-836cf8013950-config\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199335    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693a3c1f-8228-4691-bd99-f05fdf784cd4-tmp-dir\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199361    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a996538-9d06-447f-bf87-e7dbb8f078c6-tmp-dir\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199484    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f59717aa-c796-4074-b63f-2fc7e13f3633-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199536    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-daemon-config\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199587    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc5110ed-77a9-443b-b64f-5309fadadd78-signing-key\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199658    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-installation-pull-secrets\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199717    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44dtj\" (UniqueName: \"kubernetes.io/projected/c44bba77-c0b2-49cc-92c1-836cf8013950-kube-api-access-44dtj\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199764    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199768    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-installation-pull-secrets\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199794    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d543da15-0c67-4912-a39e-04235a6afa20-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.199977    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d543da15-0c67-4912-a39e-04235a6afa20-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200019    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200079    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200113    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.200227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200169    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvrb\" (UniqueName: \"kubernetes.io/projected/b1402f00-5adc-44cb-961c-a2556e51529b-kube-api-access-6lvrb\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200419    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200425    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200477    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-stats-auth\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200505    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-multus-certs\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200543    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-data-volume\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200570    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmnh\" (UniqueName: \"kubernetes.io/projected/990975fd-2466-4224-acbe-6269e49a9ef9-kube-api-access-cbmnh\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200596    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-var-lib-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200624    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200681    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200711    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvtlp\" (UniqueName: \"kubernetes.io/projected/f53b8d98-bbb7-4108-a65d-6a15ab782bee-kube-api-access-dvtlp\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200737    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cni-binary-copy\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200763    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-log-socket\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200788    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-socket-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200814    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200849    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200876    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-systemd\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.201002 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.200980    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5k6g\" (UniqueName: \"kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g\") pod \"network-check-target-pdpbv\" (UID: \"d48c64d8-b46d-48e0-830e-d9ec6a92ff06\") " pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201009    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/693a3c1f-8228-4691-bd99-f05fdf784cd4-hosts-file\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201055    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76cd3516-aff9-4de2-a063-c30407f68564-ca-trust-extracted\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201081    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-modprobe-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201105    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-run\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201134    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tz7l\" (UniqueName: \"kubernetes.io/projected/693a3c1f-8228-4691-bd99-f05fdf784cd4-kube-api-access-5tz7l\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201159    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-socket-dir-parent\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201187    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-registry-certificates\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201211    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-node-log\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201246    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-snapshots\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201271    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-kubelet\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201295    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-device-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201323    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjz6q\" (UniqueName: \"kubernetes.io/projected/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-api-access-wjz6q\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201347    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-systemd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201373    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z89wv\" (UniqueName: \"kubernetes.io/projected/dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed-kube-api-access-z89wv\") pod \"kas-connection-checker-685895c7c9-hvfpm\" (UID: \"dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed\") " pod="kube-system/kas-connection-checker-685895c7c9-hvfpm"
Oct 31 00:28:49.201491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201493    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f59717aa-c796-4074-b63f-2fc7e13f3633-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201520    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-system-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201547    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-bin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201573    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a996538-9d06-447f-bf87-e7dbb8f078c6-config-volume\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201599    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvlx\" (UniqueName: \"kubernetes.io/projected/f5703958-5751-4c4d-b979-01585c84cd46-kube-api-access-dnvlx\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201627    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201652    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-etc-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201680    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdphh\" (UniqueName: \"kubernetes.io/projected/408faab9-57a4-4c18-86ae-68772011de21-kube-api-access-mdphh\") pod \"kas-connection-checker-685895c7c9-8tt4s\" (UID: \"408faab9-57a4-4c18-86ae-68772011de21\") " pod="kube-system/kas-connection-checker-685895c7c9-8tt4s"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201705    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-lib-modules\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201732    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495pb\" (UniqueName: \"kubernetes.io/projected/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-kube-api-access-495pb\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201762    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-hostroot\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201789    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-image-registry-private-configuration\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201815    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-netns\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201842    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.201879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201870    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7lmn\" (UniqueName: \"kubernetes.io/projected/2a996538-9d06-447f-bf87-e7dbb8f078c6-kube-api-access-d7lmn\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201896    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5703958-5751-4c4d-b979-01585c84cd46-serviceca\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201922    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201949    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/990975fd-2466-4224-acbe-6269e49a9ef9-config\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.201974    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-script-lib\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202003    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-etc-selinux\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202052    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-kubernetes\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202095    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-bound-sa-token\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202122    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-tmp\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202151    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2gk\" (UniqueName: \"kubernetes.io/projected/81be0f35-6496-46d5-a27b-46bd185e3993-kube-api-access-jb2gk\") pod \"kas-connection-checker-685895c7c9-tkpbr\" (UID: \"81be0f35-6496-46d5-a27b-46bd185e3993\") " pod="kube-system/kas-connection-checker-685895c7c9-tkpbr"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202180    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-default-certificate\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202207    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-config\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202234    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovn-node-metrics-cert\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.202266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202261    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.202587 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202288    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-image-registry-private-configuration\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.202587 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202315    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db65a944-e395-4d70-9807-c061aecf0584-cert\") pod \"ingress-canary-jmdnc\" (UID: \"db65a944-e395-4d70-9807-c061aecf0584\") " pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:49.202587 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202551    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-conf\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.202587 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202580    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44wtx\" (UniqueName: \"kubernetes.io/projected/c8302492-d696-40ad-bffd-431b35319e22-kube-api-access-44wtx\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.202690 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202606    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-multus\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.202690 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202635    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b79cl\" (UniqueName: \"kubernetes.io/projected/bc77171d-558b-40c2-aa20-8e45c1693030-kube-api-access-b79cl\") pod \"migrator-66df5d496b-hbzrr\" (UID: \"bc77171d-558b-40c2-aa20-8e45c1693030\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr"
Oct 31 00:28:49.202690 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202662    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.202770 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.202692    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f8k\" (UniqueName: \"kubernetes.io/projected/0e67064c-ba5d-4780-9257-9df20f304f29-kube-api-access-r4f8k\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:49.203128 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.203109    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db65a944-e395-4d70-9807-c061aecf0584-cert\") pod \"ingress-canary-jmdnc\" (UID: \"db65a944-e395-4d70-9807-c061aecf0584\") " pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:49.203530 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.203511    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/261b0474-eb3d-4f3c-b33d-01856306e928-original-pull-secret\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:49.203740 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.203726    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-stats-auth\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.203800 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.203786    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-data-volume\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.204300 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.204262    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cni-binary-copy\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.204438 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.204419    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.204944 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.204636    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76cd3516-aff9-4de2-a063-c30407f68564-ca-trust-extracted\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.204944 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.204910    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-registry-certificates\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.205052 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.204969    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f53b8d98-bbb7-4108-a65d-6a15ab782bee-snapshots\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.205321 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.205304    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f59717aa-c796-4074-b63f-2fc7e13f3633-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:49.205479 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.205465    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a996538-9d06-447f-bf87-e7dbb8f078c6-config-volume\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:49.206098 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.206080    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-image-registry-private-configuration\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.206376 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.206358    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f53b8d98-bbb7-4108-a65d-6a15ab782bee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.206732 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.206715    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5703958-5751-4c4d-b979-01585c84cd46-serviceca\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:49.206882 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.206868    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/990975fd-2466-4224-acbe-6269e49a9ef9-config\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:49.207068 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.207055    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-script-lib\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.207356 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.207340    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8302492-d696-40ad-bffd-431b35319e22-tmp\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.207679 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.207663    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-default-certificate\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.207846 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.207831    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovnkube-config\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.207977 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.207963    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a27a3030-72b2-4223-b95e-c3997a8eb0f6-ovn-node-metrics-cert\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.208149 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.208134    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1402f00-5adc-44cb-961c-a2556e51529b-cni-binary-copy\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.208329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.208310    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-image-registry-private-configuration\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.253360 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.253330    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ppwn\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-kube-api-access-9ppwn\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.259124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.257746    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-bound-sa-token\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.259124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.257832    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9pr\" (UniqueName: \"kubernetes.io/projected/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kube-api-access-2q9pr\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.259124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.257874    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45q9\" (UniqueName: \"kubernetes.io/projected/c416b494-9d26-466e-b840-103ce32abf45-kube-api-access-j45q9\") pod \"migrator-66df5d496b-5wmv2\" (UID: \"c416b494-9d26-466e-b840-103ce32abf45\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2"
Oct 31 00:28:49.259124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.257961    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86wrt\" (UniqueName: \"kubernetes.io/projected/a27a3030-72b2-4223-b95e-c3997a8eb0f6-kube-api-access-86wrt\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.259124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.258175    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5cn\" (UniqueName: \"kubernetes.io/projected/54ecc909-648d-4848-9f49-464285458129-kube-api-access-2d5cn\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:49.259124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.258406    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvrb\" (UniqueName: \"kubernetes.io/projected/b1402f00-5adc-44cb-961c-a2556e51529b-kube-api-access-6lvrb\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.259124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.258495    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n42j6\" (UniqueName: \"kubernetes.io/projected/5266b841-4495-40ff-8277-a8c7b6fdba80-kube-api-access-n42j6\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:49.259124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.258535    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfs9t\" (UniqueName: \"kubernetes.io/projected/d7ef07d0-5483-4176-bfb2-4e03c9be4498-kube-api-access-kfs9t\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.259124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.258598    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44dtj\" (UniqueName: \"kubernetes.io/projected/c44bba77-c0b2-49cc-92c1-836cf8013950-kube-api-access-44dtj\") pod \"console-operator-7cd56b8dbc-pcbjl\" (UID: \"c44bba77-c0b2-49cc-92c1-836cf8013950\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:49.259124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.259066    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zfbv\" (UniqueName: \"kubernetes.io/projected/dc5110ed-77a9-443b-b64f-5309fadadd78-kube-api-access-9zfbv\") pod \"service-ca-5fffc97bdf-cn9rf\" (UID: \"dc5110ed-77a9-443b-b64f-5309fadadd78\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cn9rf"
Oct 31 00:28:49.259733 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.259534    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpz4f\" (UniqueName: \"kubernetes.io/projected/db65a944-e395-4d70-9807-c061aecf0584-kube-api-access-rpz4f\") pod \"ingress-canary-jmdnc\" (UID: \"db65a944-e395-4d70-9807-c061aecf0584\") " pod="openshift-ingress-canary/ingress-canary-jmdnc"
Oct 31 00:28:49.259784 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.259758    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rbld\" (UniqueName: \"kubernetes.io/projected/f59717aa-c796-4074-b63f-2fc7e13f3633-kube-api-access-5rbld\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ldsng\" (UID: \"f59717aa-c796-4074-b63f-2fc7e13f3633\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng"
Oct 31 00:28:49.259821 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.259781    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfmz7\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-kube-api-access-xfmz7\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.259851 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.259838    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czwhh\" (UniqueName: \"kubernetes.io/projected/967855d0-8720-446c-a8df-9e2b427533b3-kube-api-access-czwhh\") pod \"network-check-source-6f78d59f99-5dlfp\" (UID: \"967855d0-8720-446c-a8df-9e2b427533b3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5dlfp"
Oct 31 00:28:49.259970 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.259952    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9rt\" (UniqueName: \"kubernetes.io/projected/2e209316-e440-4bdf-b2d9-cc79ac3e670f-kube-api-access-tq9rt\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.261553 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.261532    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f8k\" (UniqueName: \"kubernetes.io/projected/0e67064c-ba5d-4780-9257-9df20f304f29-kube-api-access-r4f8k\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:49.262390 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.262365    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t88q\" (UniqueName: \"kubernetes.io/projected/34748730-f975-4008-9a37-dad912c7e930-kube-api-access-7t88q\") pod \"volume-data-source-validator-58c9f5f46d-qjwgt\" (UID: \"34748730-f975-4008-9a37-dad912c7e930\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qjwgt"
Oct 31 00:28:49.262390 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.262382    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmnh\" (UniqueName: \"kubernetes.io/projected/990975fd-2466-4224-acbe-6269e49a9ef9-kube-api-access-cbmnh\") pod \"service-ca-operator-544d749c8c-t7fd8\" (UID: \"990975fd-2466-4224-acbe-6269e49a9ef9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8"
Oct 31 00:28:49.262583 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.262563    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvtlp\" (UniqueName: \"kubernetes.io/projected/f53b8d98-bbb7-4108-a65d-6a15ab782bee-kube-api-access-dvtlp\") pod \"insights-operator-7c4986d449-4sch4\" (UID: \"f53b8d98-bbb7-4108-a65d-6a15ab782bee\") " pod="openshift-insights/insights-operator-7c4986d449-4sch4"
Oct 31 00:28:49.263688 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.263669    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7lmn\" (UniqueName: \"kubernetes.io/projected/2a996538-9d06-447f-bf87-e7dbb8f078c6-kube-api-access-d7lmn\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:49.263763 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.263692    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5k6g\" (UniqueName: \"kubernetes.io/projected/d48c64d8-b46d-48e0-830e-d9ec6a92ff06-kube-api-access-l5k6g\") pod \"network-check-target-pdpbv\" (UID: \"d48c64d8-b46d-48e0-830e-d9ec6a92ff06\") " pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:49.265445 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.265424    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-bound-sa-token\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.265798 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.265778    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tz7l\" (UniqueName: \"kubernetes.io/projected/693a3c1f-8228-4691-bd99-f05fdf784cd4-kube-api-access-5tz7l\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:49.266751 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.266728    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-495pb\" (UniqueName: \"kubernetes.io/projected/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-kube-api-access-495pb\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:49.266835 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.266825    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjz6q\" (UniqueName: \"kubernetes.io/projected/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-api-access-wjz6q\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.266959 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.266932    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z89wv\" (UniqueName: \"kubernetes.io/projected/dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed-kube-api-access-z89wv\") pod \"kas-connection-checker-685895c7c9-hvfpm\" (UID: \"dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed\") " pod="kube-system/kas-connection-checker-685895c7c9-hvfpm"
Oct 31 00:28:49.267109 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.267055    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvlx\" (UniqueName: \"kubernetes.io/projected/f5703958-5751-4c4d-b979-01585c84cd46-kube-api-access-dnvlx\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:49.267521 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.267502    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdphh\" (UniqueName: \"kubernetes.io/projected/408faab9-57a4-4c18-86ae-68772011de21-kube-api-access-mdphh\") pod \"kas-connection-checker-685895c7c9-8tt4s\" (UID: \"408faab9-57a4-4c18-86ae-68772011de21\") " pod="kube-system/kas-connection-checker-685895c7c9-8tt4s"
Oct 31 00:28:49.269292 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.269273    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b79cl\" (UniqueName: \"kubernetes.io/projected/bc77171d-558b-40c2-aa20-8e45c1693030-kube-api-access-b79cl\") pod \"migrator-66df5d496b-hbzrr\" (UID: \"bc77171d-558b-40c2-aa20-8e45c1693030\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr"
Oct 31 00:28:49.269405 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.269387    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2gk\" (UniqueName: \"kubernetes.io/projected/81be0f35-6496-46d5-a27b-46bd185e3993-kube-api-access-jb2gk\") pod \"kas-connection-checker-685895c7c9-tkpbr\" (UID: \"81be0f35-6496-46d5-a27b-46bd185e3993\") " pod="kube-system/kas-connection-checker-685895c7c9-tkpbr"
Oct 31 00:28:49.270155 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.270139    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44wtx\" (UniqueName: \"kubernetes.io/projected/c8302492-d696-40ad-bffd-431b35319e22-kube-api-access-44wtx\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.303346 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303309    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:49.303346 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303344    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-sys-fs\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303361    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-host-slash\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303386    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-kubelet\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303410    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-netd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303436    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303455    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9fad5d0-0d3a-4ebe-a07b-1899400059a7-host-slash\") pod \"iptables-alerter-8p6vl\" (UID: \"b9fad5d0-0d3a-4ebe-a07b-1899400059a7\") " pod="openshift-network-operator/iptables-alerter-8p6vl"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303463    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.303483    4935 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303494    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303484    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-netd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303459    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-sys-fs\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303491    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303459    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-kubelet\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.303556    4935 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 00:28:49.303585 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.303570    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs podName:0e67064c-ba5d-4780-9257-9df20f304f29 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.803547105 +0000 UTC m=+1.986076404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs") pod "network-metrics-daemon-7smc9" (UID: "0e67064c-ba5d-4780-9257-9df20f304f29") : secret "metrics-daemon-secret" not found
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.303597    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.803580058 +0000 UTC m=+1.986109355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : configmap references non-existent config key: service-ca.crt
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303615    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303635    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-multus-certs\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303637    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303652    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-var-lib-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.303677    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert podName:d543da15-0c67-4912-a39e-04235a6afa20 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.803661974 +0000 UTC m=+1.986191254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9lmll" (UID: "d543da15-0c67-4912-a39e-04235a6afa20") : secret "networking-console-plugin-cert" not found
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303680    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-var-lib-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303684    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-multus-certs\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303695    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303723    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303747    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-log-socket\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303757    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303769    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-socket-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303793    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303796    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303797    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-log-socket\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303821    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-systemd\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303842    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/693a3c1f-8228-4691-bd99-f05fdf784cd4-hosts-file\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.303853    4935 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303857    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-modprobe-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303870    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-run\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.303897    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls podName:2a996538-9d06-447f-bf87-e7dbb8f078c6 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.803886728 +0000 UTC m=+1.986416001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls") pod "dns-default-grc8r" (UID: "2a996538-9d06-447f-bf87-e7dbb8f078c6") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303914    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-socket-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303915    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-socket-dir-parent\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303919    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-run\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303927    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/693a3c1f-8228-4691-bd99-f05fdf784cd4-hosts-file\") pod \"node-resolver-29ctl\" (UID: \"693a3c1f-8228-4691-bd99-f05fdf784cd4\") " pod="openshift-dns/node-resolver-29ctl"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303970    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-socket-dir-parent\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.303974    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-systemd\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304017    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-node-log\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304022    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-modprobe-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304069    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-kubelet\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304095    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-device-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304099    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-kubelet\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304069    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-node-log\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304118    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-systemd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304136    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-device-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304143    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-system-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304165    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-bin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304189    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-systemd\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304191    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304231    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-bin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304237    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-system-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.304216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304232    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-etc-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304254    4935 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304260    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-etc-openvswitch\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304283    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-lib-modules\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304292    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls podName:c83cbbe5-3d68-4067-a0e9-c38f0f72df38 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.804279102 +0000 UTC m=+1.986808382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7l7z" (UID: "c83cbbe5-3d68-4067-a0e9-c38f0f72df38") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304313    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-hostroot\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304333    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-netns\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304355    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-hostroot\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304355    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304386    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-lib-modules\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304386    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-etc-selinux\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304413    4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304416    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-run-netns\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304428    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-kubernetes\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304454    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-conf\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304470    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-multus\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304413    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-etc-selinux\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304490    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304424    4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfc76855c-xm22c: secret "image-registry-tls" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304507    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-kubernetes\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304515    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304531    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-var-lib-cni-multus\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304539    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls podName:76cd3516-aff9-4de2-a063-c30407f68564 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.804526846 +0000 UTC m=+1.987056159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls") pod "image-registry-6cfc76855c-xm22c" (UID: "76cd3516-aff9-4de2-a063-c30407f68564") : secret "image-registry-tls" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304556    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-ovn\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304570    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-run-ovn\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304578    4935 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304576    4935 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304595    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-conf\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304608    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-system-cni-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304618    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.80460627 +0000 UTC m=+1.987135588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : secret "router-metrics-certs-default" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304582    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-system-cni-dir\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304650    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm podName:c83cbbe5-3d68-4067-a0e9-c38f0f72df38 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.804637857 +0000 UTC m=+1.987167130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7l7z" (UID: "c83cbbe5-3d68-4067-a0e9-c38f0f72df38") : configmap "kube-rbac-proxy" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304687    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-cnibin\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304705    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304721    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysconfig\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304730    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1402f00-5adc-44cb-961c-a2556e51529b-cnibin\") pod \"multus-additional-cni-plugins-h8q7j\" (UID: \"b1402f00-5adc-44cb-961c-a2556e51529b\") " pod="openshift-multus/multus-additional-cni-plugins-h8q7j"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304735    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cnibin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304748    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-netns\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304765    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304783    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysconfig\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304787    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-dbus\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304793    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-cnibin\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304807    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-netns\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304818    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304830    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-dbus\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304832    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-etc-sysctl-d\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304838    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-conf-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304874    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-conf-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304876    4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304893    4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f66ccc5cb-h7cvd: secret "image-registry-tls" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304901    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304887    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-multus-cni-dir\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304922    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls podName:d6511745-7315-463c-bc25-0e3e1fd6bf42 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.804911085 +0000 UTC m=+1.987440388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls") pod "image-registry-7f66ccc5cb-h7cvd" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42") : secret "image-registry-tls" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304953    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-registration-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.304965    4935 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304971    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-k8s-cni-cncf-io\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.304988    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.305000    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls podName:54ecc909-648d-4848-9f49-464285458129 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:49.804988554 +0000 UTC m=+1.987517826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r9hjn" (UID: "54ecc909-648d-4848-9f49-464285458129") : secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305022    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-systemd-units\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305031    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-host-run-k8s-cni-cncf-io\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305058    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-registration-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305069    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-bin\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305096    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-kubelet-config\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305102    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-cni-bin\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305114    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5703958-5751-4c4d-b979-01585c84cd46-host\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305114    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-systemd-units\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305135    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-sys\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305143    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/261b0474-eb3d-4f3c-b33d-01856306e928-kubelet-config\") pod \"global-pull-secret-syncer-wdzhr\" (UID: \"261b0474-eb3d-4f3c-b33d-01856306e928\") " pod="kube-system/global-pull-secret-syncer-wdzhr"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305158    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-host\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305180    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-sys\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305159    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5703958-5751-4c4d-b979-01585c84cd46-host\") pod \"node-ca-fvksv\" (UID: \"f5703958-5751-4c4d-b979-01585c84cd46\") " pod="openshift-image-registry/node-ca-fvksv"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305189    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-var-lib-kubelet\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305197    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-host\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305216    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-etc-kubernetes\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305222    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8302492-d696-40ad-bffd-431b35319e22-var-lib-kubelet\") pod \"tuned-sxssn\" (UID: \"c8302492-d696-40ad-bffd-431b35319e22\") " pod="openshift-cluster-node-tuning-operator/tuned-sxssn"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305236    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-crio-socket\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305250    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-slash\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305256    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ef07d0-5483-4176-bfb2-4e03c9be4498-etc-kubernetes\") pod \"multus-76vsc\" (UID: \"d7ef07d0-5483-4176-bfb2-4e03c9be4498\") " pod="openshift-multus/multus-76vsc"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305263    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305287    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-crio-socket\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305301    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a27a3030-72b2-4223-b95e-c3997a8eb0f6-host-slash\") pod \"ovnkube-node-z98bg\" (UID: \"a27a3030-72b2-4223-b95e-c3997a8eb0f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305348    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34d3cbf5-134e-4f86-8b33-04ef38cdb685-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n649s\" (UID: \"34d3cbf5-134e-4f86-8b33-04ef38cdb685\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n649s"
Oct 31 00:28:49.306107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.305342    4935 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:28:49.309646 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.307719    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5266b841-4495-40ff-8277-a8c7b6fdba80-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9h6hk\" (UID: \"5266b841-4495-40ff-8277-a8c7b6fdba80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:49.337788 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.337756    4935 scope.go:122] "RemoveContainer" containerID="1010115f821e74be8029255505b4165efa642f8bce3ec324faae93519f703f55"
Oct 31 00:28:49.338119 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.338081    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"
Oct 31 00:28:49.355138 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.355097    4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:28:49.442464 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.442430    4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:49.720564 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.720479    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk"]
Oct 31 00:28:49.759771 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.759743    4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:49.809622 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.809595    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.809756 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.809635    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.809756 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.809673    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:49.809756 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.809706    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:49.809756 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.809711    4935 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.809767    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm podName:c83cbbe5-3d68-4067-a0e9-c38f0f72df38 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.809752638 +0000 UTC m=+2.992281947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x7l7z" (UID: "c83cbbe5-3d68-4067-a0e9-c38f0f72df38") : configmap "kube-rbac-proxy" not found
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.809786    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.809802    4935 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.809808    4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.809817    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.809833    4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f66ccc5cb-h7cvd: secret "image-registry-tls" not found
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.809858    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.809842771 +0000 UTC m=+2.992372075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : secret "router-metrics-certs-default" not found
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.809815    4935 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.809876    4935 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.809904    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.809868868 +0000 UTC m=+2.992398142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : configmap references non-existent config key: service-ca.crt
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.809924    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls podName:d6511745-7315-463c-bc25-0e3e1fd6bf42 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.809916684 +0000 UTC m=+2.992445956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls") pod "image-registry-7f66ccc5cb-h7cvd" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42") : secret "image-registry-tls" not found
Oct 31 00:28:49.809956 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.809942    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.809975    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.809981    4935 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.810003    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert podName:d543da15-0c67-4912-a39e-04235a6afa20 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.809997234 +0000 UTC m=+2.992526549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9lmll" (UID: "d543da15-0c67-4912-a39e-04235a6afa20") : secret "networking-console-plugin-cert" not found
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.810026    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls podName:54ecc909-648d-4848-9f49-464285458129 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.810015434 +0000 UTC m=+2.992544707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r9hjn" (UID: "54ecc909-648d-4848-9f49-464285458129") : secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.810053    4935 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.810067    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs podName:0e67064c-ba5d-4780-9257-9df20f304f29 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.810060292 +0000 UTC m=+2.992589564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs") pod "network-metrics-daemon-7smc9" (UID: "0e67064c-ba5d-4780-9257-9df20f304f29") : secret "metrics-daemon-secret" not found
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.810083    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.810088    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls podName:2a996538-9d06-447f-bf87-e7dbb8f078c6 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.810079021 +0000 UTC m=+2.992608310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls") pod "dns-default-grc8r" (UID: "2a996538-9d06-447f-bf87-e7dbb8f078c6") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.810131    4935 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.810155    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls podName:c83cbbe5-3d68-4067-a0e9-c38f0f72df38 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.810148511 +0000 UTC m=+2.992677783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7l7z" (UID: "c83cbbe5-3d68-4067-a0e9-c38f0f72df38") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:49.810171    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.810238    4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.810245    4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfc76855c-xm22c: secret "image-registry-tls" not found
Oct 31 00:28:49.810404 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:49.810270    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls podName:76cd3516-aff9-4de2-a063-c30407f68564 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:50.810264326 +0000 UTC m=+2.992793598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls") pod "image-registry-6cfc76855c-xm22c" (UID: "76cd3516-aff9-4de2-a063-c30407f68564") : secret "image-registry-tls" not found
Oct 31 00:28:50.005674 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.005638    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk" event={"ID":"5266b841-4495-40ff-8277-a8c7b6fdba80","Type":"ContainerStarted","Data":"bb9c02e3c91165272a9bf2460982aa60b3cfbe2f6d77049f88e3ffcc517f5903"}
Oct 31 00:28:50.007235 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.007217    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:28:50.007606 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.007590    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/1.log"
Oct 31 00:28:50.007693 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.007626    4935 generic.go:356] "Generic (PLEG): container finished" podID="c44bba77-c0b2-49cc-92c1-836cf8013950" containerID="f41f352a6c06c5e3136fdc46465ad72843186757652cc7de2db37d3b080dd0f8" exitCode=255
Oct 31 00:28:50.007750 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.007703    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" event={"ID":"c44bba77-c0b2-49cc-92c1-836cf8013950","Type":"ContainerDied","Data":"f41f352a6c06c5e3136fdc46465ad72843186757652cc7de2db37d3b080dd0f8"}
Oct 31 00:28:50.007799 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.007755    4935 scope.go:122] "RemoveContainer" containerID="1010115f821e74be8029255505b4165efa642f8bce3ec324faae93519f703f55"
Oct 31 00:28:50.008323 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.008304    4935 scope.go:122] "RemoveContainer" containerID="f41f352a6c06c5e3136fdc46465ad72843186757652cc7de2db37d3b080dd0f8"
Oct 31 00:28:50.008471 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.008454    4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcbjl_openshift-console-operator(c44bba77-c0b2-49cc-92c1-836cf8013950)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" podUID="c44bba77-c0b2-49cc-92c1-836cf8013950"
Oct 31 00:28:50.168436 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.168396    4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:50.367912 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.367828    4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:50.587873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.587836    4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:50.594427 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.594397    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:50.819676 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.819613    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:50.819676 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.819675    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:50.819852 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.819709    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:50.819852 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.819746    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:50.819852 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.819788    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:50.819852 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.819822    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:50.819980 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.819858    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:50.819980 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.819882    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:50.819980 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.819915    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:50.819980 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.819944    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:50.820166 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.820141    4935 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:50.820255 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.820209    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls podName:54ecc909-648d-4848-9f49-464285458129 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:52.820189282 +0000 UTC m=+5.002718572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r9hjn" (UID: "54ecc909-648d-4848-9f49-464285458129") : secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:50.820628 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.820613    4935 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:28:50.820708 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.820661    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs podName:0e67064c-ba5d-4780-9257-9df20f304f29 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:52.820646821 +0000 UTC m=+5.003176107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs") pod "network-metrics-daemon-7smc9" (UID: "0e67064c-ba5d-4780-9257-9df20f304f29") : secret "metrics-daemon-secret" not found
Oct 31 00:28:50.820765 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.820732    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:52.820721634 +0000 UTC m=+5.003250920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : configmap references non-existent config key: service-ca.crt
Oct 31 00:28:50.820807 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.820790    4935 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 00:28:50.820852 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.820820    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert podName:d543da15-0c67-4912-a39e-04235a6afa20 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:52.820809969 +0000 UTC m=+5.003339255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9lmll" (UID: "d543da15-0c67-4912-a39e-04235a6afa20") : secret "networking-console-plugin-cert" not found
Oct 31 00:28:50.820889 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.820881    4935 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:28:50.820917 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.820909    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls podName:2a996538-9d06-447f-bf87-e7dbb8f078c6 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:52.820899874 +0000 UTC m=+5.003429159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls") pod "dns-default-grc8r" (UID: "2a996538-9d06-447f-bf87-e7dbb8f078c6") : secret "dns-default-metrics-tls" not found
Oct 31 00:28:50.820982 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.820965    4935 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:50.821062 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.820998    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls podName:c83cbbe5-3d68-4067-a0e9-c38f0f72df38 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:52.820986872 +0000 UTC m=+5.003516158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7l7z" (UID: "c83cbbe5-3d68-4067-a0e9-c38f0f72df38") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:50.821098 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.821075    4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:50.821098 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.821086    4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfc76855c-xm22c: secret "image-registry-tls" not found
Oct 31 00:28:50.821187 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.821117    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls podName:76cd3516-aff9-4de2-a063-c30407f68564 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:52.821106885 +0000 UTC m=+5.003636172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls") pod "image-registry-6cfc76855c-xm22c" (UID: "76cd3516-aff9-4de2-a063-c30407f68564") : secret "image-registry-tls" not found
Oct 31 00:28:50.821687 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.821665    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:50.821783 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.821771    4935 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 00:28:50.821825 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.821817    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:52.821802365 +0000 UTC m=+5.004331651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : secret "router-metrics-certs-default" not found
Oct 31 00:28:50.821891 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.821881    4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:50.821937 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.821896    4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f66ccc5cb-h7cvd: secret "image-registry-tls" not found
Oct 31 00:28:50.821937 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:50.821925    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls podName:d6511745-7315-463c-bc25-0e3e1fd6bf42 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:52.821914988 +0000 UTC m=+5.004444275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls") pod "image-registry-7f66ccc5cb-h7cvd" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42") : secret "image-registry-tls" not found
Oct 31 00:28:50.964307 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:50.964273    4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:51.015248 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:51.015213    4935 generic.go:356] "Generic (PLEG): container finished" podID="f53b8d98-bbb7-4108-a65d-6a15ab782bee" containerID="b81900eec2c2405e13a75b226b2fd7840b8cabb06ac2c5be5d7e034aac4b28e6" exitCode=0
Oct 31 00:28:51.015712 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:51.015256    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4sch4" event={"ID":"f53b8d98-bbb7-4108-a65d-6a15ab782bee","Type":"ContainerDied","Data":"b81900eec2c2405e13a75b226b2fd7840b8cabb06ac2c5be5d7e034aac4b28e6"}
Oct 31 00:28:51.015770 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:51.015743    4935 scope.go:122] "RemoveContainer" containerID="b81900eec2c2405e13a75b226b2fd7840b8cabb06ac2c5be5d7e034aac4b28e6"
Oct 31 00:28:51.018012 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:51.017959    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:28:51.018329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:51.018313    4935 scope.go:122] "RemoveContainer" containerID="f41f352a6c06c5e3136fdc46465ad72843186757652cc7de2db37d3b080dd0f8"
Oct 31 00:28:51.018542 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:51.018519    4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcbjl_openshift-console-operator(c44bba77-c0b2-49cc-92c1-836cf8013950)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" podUID="c44bba77-c0b2-49cc-92c1-836cf8013950"
Oct 31 00:28:51.193975 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:51.193947    4935 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log" limit=1024
Oct 31 00:28:51.495204 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:51.495172    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:52.022939 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.022903    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hbzrr" event={"ID":"bc77171d-558b-40c2-aa20-8e45c1693030","Type":"ContainerStarted","Data":"dfc04973d5327bc393fc76e52daabcfbaba18f557bfa0c5c76dce5b1c8f3cc6a"}
Oct 31 00:28:52.024652 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.024626    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5wmv2" event={"ID":"c416b494-9d26-466e-b840-103ce32abf45","Type":"ContainerStarted","Data":"bd1780ef28056d53949cf5920c19f1ac4905b8d0df7bd547cf6d2121a8f3ab87"}
Oct 31 00:28:52.026764 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.026735    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4sch4" event={"ID":"f53b8d98-bbb7-4108-a65d-6a15ab782bee","Type":"ContainerStarted","Data":"8ab125f20dccff9382c944f4b13b771a64ca6b8bd5e251fa3b4ce1bd10024a56"}
Oct 31 00:28:52.842627 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.842586    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:52.842627 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.842630    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:52.842877 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.842665    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:52.842877 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.842693    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:52.842877 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.842719    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:52.842877 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.842755    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:52.842877 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842763    4935 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 00:28:52.842877 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.842787    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:52.842877 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.842819    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:52.842877 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842840    4935 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:52.842877 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842846    4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:52.842877 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842867    4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cfc76855c-xm22c: secret "image-registry-tls" not found
Oct 31 00:28:52.842877 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842879    4935 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:52.843285 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842881    4935 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 00:28:52.843285 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842844    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert podName:d543da15-0c67-4912-a39e-04235a6afa20 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:56.842821914 +0000 UTC m=+9.025351190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9lmll" (UID: "d543da15-0c67-4912-a39e-04235a6afa20") : secret "networking-console-plugin-cert" not found
Oct 31 00:28:52.843285 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842936    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls podName:54ecc909-648d-4848-9f49-464285458129 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:56.842908209 +0000 UTC m=+9.025437485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r9hjn" (UID: "54ecc909-648d-4848-9f49-464285458129") : secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:52.843285 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842955    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls podName:c83cbbe5-3d68-4067-a0e9-c38f0f72df38 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:56.842945753 +0000 UTC m=+9.025475025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7l7z" (UID: "c83cbbe5-3d68-4067-a0e9-c38f0f72df38") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:52.843285 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842959    4935 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 00:28:52.843285 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842971    4935 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f66ccc5cb-h7cvd: secret "image-registry-tls" not found
Oct 31 00:28:52.843285 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.842977    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls podName:76cd3516-aff9-4de2-a063-c30407f68564 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:56.842966266 +0000 UTC m=+9.025495539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls") pod "image-registry-6cfc76855c-xm22c" (UID: "76cd3516-aff9-4de2-a063-c30407f68564") : secret "image-registry-tls" not found
Oct 31 00:28:52.843285 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.843001    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs podName:2e209316-e440-4bdf-b2d9-cc79ac3e670f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:56.842992606 +0000 UTC m=+9.025521885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs") pod "router-default-55c864699c-n4cfn" (UID: "2e209316-e440-4bdf-b2d9-cc79ac3e670f") : secret "router-metrics-certs-default" not found
Oct 31 00:28:52.843285 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.843019    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls podName:d6511745-7315-463c-bc25-0e3e1fd6bf42 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:56.843011432 +0000 UTC m=+9.025540709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls") pod "image-registry-7f66ccc5cb-h7cvd" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42") : secret "image-registry-tls" not found
Oct 31 00:28:52.843285 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.843105    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:52.843285 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.843242    4935 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:28:52.843601 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:52.843300    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs podName:0e67064c-ba5d-4780-9257-9df20f304f29 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:56.843288346 +0000 UTC m=+9.025817619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs") pod "network-metrics-daemon-7smc9" (UID: "0e67064c-ba5d-4780-9257-9df20f304f29") : secret "metrics-daemon-secret" not found
Oct 31 00:28:52.843601 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.843361    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e209316-e440-4bdf-b2d9-cc79ac3e670f-service-ca-bundle\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:52.845211 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.845194    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a996538-9d06-447f-bf87-e7dbb8f078c6-metrics-tls\") pod \"dns-default-grc8r\" (UID: \"2a996538-9d06-447f-bf87-e7dbb8f078c6\") " pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:52.953063 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:52.953005    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:53.033360 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:53.033325    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk" event={"ID":"5266b841-4495-40ff-8277-a8c7b6fdba80","Type":"ContainerStarted","Data":"f549a8bf97b24a7e501dd7e4bdf2cd8e97a1ef3ab76e4f12c60fbc65814ffc3a"}
Oct 31 00:28:53.033821 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:53.033390    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9h6hk" event={"ID":"5266b841-4495-40ff-8277-a8c7b6fdba80","Type":"ContainerStarted","Data":"21b9a197c835fbd06b8780ac830c7780ee25511544d8c06b03f8675e3ca2a647"}
Oct 31 00:28:53.325310 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:53.325276    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-grc8r"]
Oct 31 00:28:53.332237 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:53.332207    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a996538_9d06_447f_bf87_e7dbb8f078c6.slice/crio-f032c2914a09f495e7ec2e80f632a3c9d406a18bc36fc0f5134855c1d785e11f WatchSource:0}: Error finding container f032c2914a09f495e7ec2e80f632a3c9d406a18bc36fc0f5134855c1d785e11f: Status 404 returned error can't find the container with id f032c2914a09f495e7ec2e80f632a3c9d406a18bc36fc0f5134855c1d785e11f
Oct 31 00:28:54.037431 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:54.037399    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-grc8r" event={"ID":"2a996538-9d06-447f-bf87-e7dbb8f078c6","Type":"ContainerStarted","Data":"f032c2914a09f495e7ec2e80f632a3c9d406a18bc36fc0f5134855c1d785e11f"}
Oct 31 00:28:55.771908 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:55.771884    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:28:55.931731 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:55.931694    4935 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:28:55.932150 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:55.932138    4935 scope.go:122] "RemoveContainer" containerID="f41f352a6c06c5e3136fdc46465ad72843186757652cc7de2db37d3b080dd0f8"
Oct 31 00:28:55.932334 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:55.932316    4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pcbjl_openshift-console-operator(c44bba77-c0b2-49cc-92c1-836cf8013950)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" podUID="c44bba77-c0b2-49cc-92c1-836cf8013950"
Oct 31 00:28:56.047906 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.047873    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-grc8r" event={"ID":"2a996538-9d06-447f-bf87-e7dbb8f078c6","Type":"ContainerStarted","Data":"6433dac1add2695e8271e5904ae7b8c13dcd305360bef382c669ff6004d61c15"}
Oct 31 00:28:56.048072 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.047927    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-grc8r" event={"ID":"2a996538-9d06-447f-bf87-e7dbb8f078c6","Type":"ContainerStarted","Data":"4f0c4bbb0f010faae294c8fce3f0fbd163ab8e85830bdfa9d7ac0fe8deaaa527"}
Oct 31 00:28:56.048072 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.047966    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-grc8r"
Oct 31 00:28:56.879010 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.878967    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:56.879010 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.879013    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:56.879873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.879050    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:28:56.879873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.879097    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:56.879873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.879126    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:56.879873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.879179    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:28:56.879873 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:56.879209    4935 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:56.879873 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:56.879278    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls podName:54ecc909-648d-4848-9f49-464285458129 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:04.879260447 +0000 UTC m=+17.061789724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-r9hjn" (UID: "54ecc909-648d-4848-9f49-464285458129") : secret "cluster-monitoring-operator-tls" not found
Oct 31 00:28:56.879873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.879211    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:56.879873 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:56.879713    4935 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:56.879873 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:28:56.879781    4935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls podName:c83cbbe5-3d68-4067-a0e9-c38f0f72df38 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:04.879760009 +0000 UTC m=+17.062289286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x7l7z" (UID: "c83cbbe5-3d68-4067-a0e9-c38f0f72df38") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:28:56.882052 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.882017    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209316-e440-4bdf-b2d9-cc79ac3e670f-metrics-certs\") pod \"router-default-55c864699c-n4cfn\" (UID: \"2e209316-e440-4bdf-b2d9-cc79ac3e670f\") " pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:56.882199 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.882180    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e67064c-ba5d-4780-9257-9df20f304f29-metrics-certs\") pod \"network-metrics-daemon-7smc9\" (UID: \"0e67064c-ba5d-4780-9257-9df20f304f29\") " pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:56.882276 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.882232    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"image-registry-7f66ccc5cb-h7cvd\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") " pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:56.882387 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.882368    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"image-registry-6cfc76855c-xm22c\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") " pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:56.882659 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:56.882641    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d543da15-0c67-4912-a39e-04235a6afa20-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9lmll\" (UID: \"d543da15-0c67-4912-a39e-04235a6afa20\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:57.137585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.137496    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:28:57.138566 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.138541    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:57.140650 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.140635    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll"
Oct 31 00:28:57.147210 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.147187    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7smc9"
Oct 31 00:28:57.151424 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.151404    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:28:57.583521 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.583487    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9lmll"]
Oct 31 00:28:57.590743 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:57.590717    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd543da15_0c67_4912_a39e_04235a6afa20.slice/crio-597f14be68cd3a801f0d153ef1e29447bef08970ffcfee8f0653e5e911ebf58d WatchSource:0}: Error finding container 597f14be68cd3a801f0d153ef1e29447bef08970ffcfee8f0653e5e911ebf58d: Status 404 returned error can't find the container with id 597f14be68cd3a801f0d153ef1e29447bef08970ffcfee8f0653e5e911ebf58d
Oct 31 00:28:57.638078 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.638030    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7smc9"]
Oct 31 00:28:57.642135 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.642109    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"]
Oct 31 00:28:57.642727 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.642669    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55c864699c-n4cfn"]
Oct 31 00:28:57.645476 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:57.645451    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e67064c_ba5d_4780_9257_9df20f304f29.slice/crio-5c3185391f3716031d856d269731b50fa02cdc27be3ec42d181db183b6180efd WatchSource:0}: Error finding container 5c3185391f3716031d856d269731b50fa02cdc27be3ec42d181db183b6180efd: Status 404 returned error can't find the container with id 5c3185391f3716031d856d269731b50fa02cdc27be3ec42d181db183b6180efd
Oct 31 00:28:57.648635 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:57.648616    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e209316_e440_4bdf_b2d9_cc79ac3e670f.slice/crio-4326e51b0f58129bfaa2f6f4e894f11892ba1cd7ccf940c0ecb4cb7fb7874254 WatchSource:0}: Error finding container 4326e51b0f58129bfaa2f6f4e894f11892ba1cd7ccf940c0ecb4cb7fb7874254: Status 404 returned error can't find the container with id 4326e51b0f58129bfaa2f6f4e894f11892ba1cd7ccf940c0ecb4cb7fb7874254
Oct 31 00:28:57.649402 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:57.649379    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6511745_7315_463c_bc25_0e3e1fd6bf42.slice/crio-5d7d9e9ed4b036e947f1ccddc9156f3ab071a089d925f1e1c3782aa047ff4c32 WatchSource:0}: Error finding container 5d7d9e9ed4b036e947f1ccddc9156f3ab071a089d925f1e1c3782aa047ff4c32: Status 404 returned error can't find the container with id 5d7d9e9ed4b036e947f1ccddc9156f3ab071a089d925f1e1c3782aa047ff4c32
Oct 31 00:28:57.694399 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.694372    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cfc76855c-xm22c"]
Oct 31 00:28:57.702593 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:28:57.702566    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cd3516_aff9_4de2_a063_c30407f68564.slice/crio-d4dbc08c75fc4b8c82f55fc48380d0fc57837ed28e2c1d919531492e692b2a3d WatchSource:0}: Error finding container d4dbc08c75fc4b8c82f55fc48380d0fc57837ed28e2c1d919531492e692b2a3d: Status 404 returned error can't find the container with id d4dbc08c75fc4b8c82f55fc48380d0fc57837ed28e2c1d919531492e692b2a3d
Oct 31 00:28:57.748810 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.748620    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:28:57.750463 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:57.749123    4935 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:28:58.057310 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:58.057268    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55c864699c-n4cfn" event={"ID":"2e209316-e440-4bdf-b2d9-cc79ac3e670f","Type":"ContainerStarted","Data":"77ef4462600dcb4a0afec50543aebbbb9f3d06a3de10e178d5a3086c55f39a7c"}
Oct 31 00:28:58.057674 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:58.057332    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55c864699c-n4cfn" event={"ID":"2e209316-e440-4bdf-b2d9-cc79ac3e670f","Type":"ContainerStarted","Data":"4326e51b0f58129bfaa2f6f4e894f11892ba1cd7ccf940c0ecb4cb7fb7874254"}
Oct 31 00:28:58.060537 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:58.060506    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c" event={"ID":"76cd3516-aff9-4de2-a063-c30407f68564","Type":"ContainerStarted","Data":"aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc"}
Oct 31 00:28:58.060726 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:58.060708    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c" event={"ID":"76cd3516-aff9-4de2-a063-c30407f68564","Type":"ContainerStarted","Data":"d4dbc08c75fc4b8c82f55fc48380d0fc57837ed28e2c1d919531492e692b2a3d"}
Oct 31 00:28:58.062099 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:58.062064    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll" event={"ID":"d543da15-0c67-4912-a39e-04235a6afa20","Type":"ContainerStarted","Data":"597f14be68cd3a801f0d153ef1e29447bef08970ffcfee8f0653e5e911ebf58d"}
Oct 31 00:28:58.064025 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:58.063999    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd" event={"ID":"d6511745-7315-463c-bc25-0e3e1fd6bf42","Type":"ContainerStarted","Data":"9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7"}
Oct 31 00:28:58.064186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:58.064161    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd" event={"ID":"d6511745-7315-463c-bc25-0e3e1fd6bf42","Type":"ContainerStarted","Data":"5d7d9e9ed4b036e947f1ccddc9156f3ab071a089d925f1e1c3782aa047ff4c32"}
Oct 31 00:28:58.065364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:58.065342    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7smc9" event={"ID":"0e67064c-ba5d-4780-9257-9df20f304f29","Type":"ContainerStarted","Data":"5c3185391f3716031d856d269731b50fa02cdc27be3ec42d181db183b6180efd"}
Oct 31 00:28:58.139263 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:58.139231    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:28:58.142957 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:58.142918    4935 patch_prober.go:29] interesting pod/router-default-55c864699c-n4cfn container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Oct 31 00:28:58.142957 ip-10-0-132-125 kubenswrapper[4935]: [-]has-synced failed: reason withheld
Oct 31 00:28:58.142957 ip-10-0-132-125 kubenswrapper[4935]: [+]process-running ok
Oct 31 00:28:58.142957 ip-10-0-132-125 kubenswrapper[4935]: healthz check failed
Oct 31 00:28:58.143213 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:58.142984    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-55c864699c-n4cfn" podUID="2e209316-e440-4bdf-b2d9-cc79ac3e670f" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Oct 31 00:28:59.069435 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:59.069399    4935 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:28:59.069435 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:59.069399    4935 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:28:59.140432 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:28:59.140395    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:29:00.074059 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:00.074005    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9lmll" event={"ID":"d543da15-0c67-4912-a39e-04235a6afa20","Type":"ContainerStarted","Data":"b932488202cc514e64b5867d2cc9a5ebf712a797642fe20605deca0208384ade"}
Oct 31 00:29:00.075822 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:00.075797    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7smc9" event={"ID":"0e67064c-ba5d-4780-9257-9df20f304f29","Type":"ContainerStarted","Data":"861fcbd0a25339678302f0bc5fd38011323567b9fd314fab3d6eeb68a4e71940"}
Oct 31 00:29:00.075959 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:00.075848    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7smc9" event={"ID":"0e67064c-ba5d-4780-9257-9df20f304f29","Type":"ContainerStarted","Data":"bde903bb9236e5f89bc0b73d7b257582b24c6acc9324d358879cc7bb8b9a0d13"}
Oct 31 00:29:00.076071 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:00.076055    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:29:00.077201 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:00.077185    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-55c864699c-n4cfn"
Oct 31 00:29:00.937962 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:00.937928    4935 scope.go:122] "RemoveContainer" containerID="f41f352a6c06c5e3136fdc46465ad72843186757652cc7de2db37d3b080dd0f8"
Oct 31 00:29:01.081790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:01.081765    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:29:01.082209 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:01.081881    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" event={"ID":"c44bba77-c0b2-49cc-92c1-836cf8013950","Type":"ContainerStarted","Data":"23271f58b12772403689b01106ef60927c46bea0c44a7524b44084ce83aa3574"}
Oct 31 00:29:01.082479 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:01.082457    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:29:01.083453 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:01.083434    4935 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-pcbjl container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body=
Oct 31 00:29:01.083540 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:01.083468    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl" podUID="c44bba77-c0b2-49cc-92c1-836cf8013950" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused"
Oct 31 00:29:01.548190 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:01.548143    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:29:02.089885 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:02.089854    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pcbjl"
Oct 31 00:29:04.720552 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:04.720514    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:29:04.958486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:04.958444    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:29:04.958698 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:04.958543    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:29:04.960989 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:04.960957    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c83cbbe5-3d68-4067-a0e9-c38f0f72df38-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7l7z\" (UID: \"c83cbbe5-3d68-4067-a0e9-c38f0f72df38\") " pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:29:04.961128 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:04.961007    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/54ecc909-648d-4848-9f49-464285458129-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-r9hjn\" (UID: \"54ecc909-648d-4848-9f49-464285458129\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:29:05.239625 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:05.239598    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"
Oct 31 00:29:05.258308 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:05.258253    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x7l7z"
Oct 31 00:29:05.669545 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:05.669467    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn"]
Oct 31 00:29:05.676391 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:29:05.676365    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ecc909_648d_4848_9f49_464285458129.slice/crio-14f5334dc1c90cbadd5b98e9f60381a85e30af4bd9f26df00bdf87fe5650608b WatchSource:0}: Error finding container 14f5334dc1c90cbadd5b98e9f60381a85e30af4bd9f26df00bdf87fe5650608b: Status 404 returned error can't find the container with id 14f5334dc1c90cbadd5b98e9f60381a85e30af4bd9f26df00bdf87fe5650608b
Oct 31 00:29:05.714794 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:29:05.714764    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83cbbe5_3d68_4067_a0e9_c38f0f72df38.slice/crio-09140522bb86af55569e48fa81c6c8736599eaf3e5b935a154c5948d7385ca63 WatchSource:0}: Error finding container 09140522bb86af55569e48fa81c6c8736599eaf3e5b935a154c5948d7385ca63: Status 404 returned error can't find the container with id 09140522bb86af55569e48fa81c6c8736599eaf3e5b935a154c5948d7385ca63
Oct 31 00:29:05.715980 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:05.715956    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x7l7z"]
Oct 31 00:29:06.032261 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:06.032235    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z98bg"
Oct 31 00:29:06.053859 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:06.053827    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-grc8r"
Oct 31 00:29:06.102438 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:06.102403    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn" event={"ID":"54ecc909-648d-4848-9f49-464285458129","Type":"ContainerStarted","Data":"14f5334dc1c90cbadd5b98e9f60381a85e30af4bd9f26df00bdf87fe5650608b"}
Oct 31 00:29:06.103858 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:06.103833    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7l7z" event={"ID":"c83cbbe5-3d68-4067-a0e9-c38f0f72df38","Type":"ContainerStarted","Data":"7ffb6a5f72d2511b84e2e1e00fed3bb22c56d61a6c97a433b18f72b166fbdafa"}
Oct 31 00:29:06.103944 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:06.103881    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7l7z" event={"ID":"c83cbbe5-3d68-4067-a0e9-c38f0f72df38","Type":"ContainerStarted","Data":"09140522bb86af55569e48fa81c6c8736599eaf3e5b935a154c5948d7385ca63"}
Oct 31 00:29:07.500274 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.499777    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-xf77w-4cxng"]
Oct 31 00:29:07.503687 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.503660    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.519685 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.519658    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-qtfnw-lnbrj"]
Oct 31 00:29:07.523066 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.523025    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.544607 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.544571    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-c7h2h"]
Oct 31 00:29:07.548345 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.548317    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-c7h2h"
Oct 31 00:29:07.566054 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.566011    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"]
Oct 31 00:29:07.579539 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.579426    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-archives-path\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.579539 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.579476    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6tc5\" (UniqueName: \"kubernetes.io/projected/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-kube-api-access-q6tc5\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.579766 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.579590    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-service-ca-bundle\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.579766 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.579624    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-trusted-ca-bundle\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.579766 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.579647    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-archives-path\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.579766 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.579704    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kfj6\" (UniqueName: \"kubernetes.io/projected/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-kube-api-access-6kfj6\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.579766 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.579753    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-service-ca-bundle\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.579988 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.579843    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-trusted-ca-bundle\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.590614 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.590577    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xf77w-4cxng"]
Oct 31 00:29:07.601371 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.601339    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 00:29:07.601371 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.601363    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 00:29:07.608197 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.608169    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-c7h2h"]
Oct 31 00:29:07.609911 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.609890    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-t62xk\""
Oct 31 00:29:07.621435 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.621361    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qtfnw-lnbrj"]
Oct 31 00:29:07.653703 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.653667    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c85cc4984-dfrbl"]
Oct 31 00:29:07.657400 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.657364    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.680438 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680410    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-archives-path\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.680613 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680445    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6tc5\" (UniqueName: \"kubernetes.io/projected/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-kube-api-access-q6tc5\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.680613 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680466    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-service-ca-bundle\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.680613 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680482    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-trusted-ca-bundle\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.680613 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680506    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-archives-path\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.680613 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680546    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kfj6\" (UniqueName: \"kubernetes.io/projected/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-kube-api-access-6kfj6\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.680613 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680565    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-service-ca-bundle\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.680971 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680621    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xp5b\" (UniqueName: \"kubernetes.io/projected/bc1d53d2-3d16-4d6f-97f5-e6c823857e4d-kube-api-access-2xp5b\") pod \"downloads-6dd789c8b7-c7h2h\" (UID: \"bc1d53d2-3d16-4d6f-97f5-e6c823857e4d\") " pod="openshift-console/downloads-6dd789c8b7-c7h2h"
Oct 31 00:29:07.680971 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680645    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-trusted-ca-bundle\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.680971 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680691    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc1d53d2-3d16-4d6f-97f5-e6c823857e4d-tmp\") pod \"downloads-6dd789c8b7-c7h2h\" (UID: \"bc1d53d2-3d16-4d6f-97f5-e6c823857e4d\") " pod="openshift-console/downloads-6dd789c8b7-c7h2h"
Oct 31 00:29:07.680971 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680884    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-archives-path\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.680971 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.680908    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-archives-path\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.681341 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.681317    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-service-ca-bundle\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.681477 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.681451    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-service-ca-bundle\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.681619 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.681599    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-trusted-ca-bundle\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.681661 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.681599    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-trusted-ca-bundle\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.724150 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.724081    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c85cc4984-dfrbl"]
Oct 31 00:29:07.741277 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.741245    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kfj6\" (UniqueName: \"kubernetes.io/projected/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-kube-api-access-6kfj6\") pod \"periodic-gathering-qtfnw-lnbrj\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") " pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.741517 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.741498    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6tc5\" (UniqueName: \"kubernetes.io/projected/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-kube-api-access-q6tc5\") pod \"periodic-gathering-xf77w-4cxng\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") " pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.781663 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.781567    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-944tk\" (UniqueName: \"kubernetes.io/projected/56566f0f-9187-433c-bc3f-5b6f6c04d88f-kube-api-access-944tk\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.781663 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.781633    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56566f0f-9187-433c-bc3f-5b6f6c04d88f-trusted-ca\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.781945 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.781715    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56566f0f-9187-433c-bc3f-5b6f6c04d88f-installation-pull-secrets\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.781945 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.781848    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56566f0f-9187-433c-bc3f-5b6f6c04d88f-bound-sa-token\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.781945 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.781871    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xp5b\" (UniqueName: \"kubernetes.io/projected/bc1d53d2-3d16-4d6f-97f5-e6c823857e4d-kube-api-access-2xp5b\") pod \"downloads-6dd789c8b7-c7h2h\" (UID: \"bc1d53d2-3d16-4d6f-97f5-e6c823857e4d\") " pod="openshift-console/downloads-6dd789c8b7-c7h2h"
Oct 31 00:29:07.781945 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.781895    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56566f0f-9187-433c-bc3f-5b6f6c04d88f-ca-trust-extracted\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.782263 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.781955    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56566f0f-9187-433c-bc3f-5b6f6c04d88f-image-registry-private-configuration\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.782263 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.781999    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc1d53d2-3d16-4d6f-97f5-e6c823857e4d-tmp\") pod \"downloads-6dd789c8b7-c7h2h\" (UID: \"bc1d53d2-3d16-4d6f-97f5-e6c823857e4d\") " pod="openshift-console/downloads-6dd789c8b7-c7h2h"
Oct 31 00:29:07.782263 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.782071    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56566f0f-9187-433c-bc3f-5b6f6c04d88f-registry-tls\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.782263 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.782106    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56566f0f-9187-433c-bc3f-5b6f6c04d88f-registry-certificates\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.782497 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.782480    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc1d53d2-3d16-4d6f-97f5-e6c823857e4d-tmp\") pod \"downloads-6dd789c8b7-c7h2h\" (UID: \"bc1d53d2-3d16-4d6f-97f5-e6c823857e4d\") " pod="openshift-console/downloads-6dd789c8b7-c7h2h"
Oct 31 00:29:07.827072 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.827016    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:29:07.837188 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.837160    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:29:07.843327 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.843306    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xp5b\" (UniqueName: \"kubernetes.io/projected/bc1d53d2-3d16-4d6f-97f5-e6c823857e4d-kube-api-access-2xp5b\") pod \"downloads-6dd789c8b7-c7h2h\" (UID: \"bc1d53d2-3d16-4d6f-97f5-e6c823857e4d\") " pod="openshift-console/downloads-6dd789c8b7-c7h2h"
Oct 31 00:29:07.863342 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.863048    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-c7h2h"
Oct 31 00:29:07.883104 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.883077    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56566f0f-9187-433c-bc3f-5b6f6c04d88f-registry-tls\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.883260 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.883122    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56566f0f-9187-433c-bc3f-5b6f6c04d88f-registry-certificates\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.883260 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.883169    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-944tk\" (UniqueName: \"kubernetes.io/projected/56566f0f-9187-433c-bc3f-5b6f6c04d88f-kube-api-access-944tk\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.883260 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.883204    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56566f0f-9187-433c-bc3f-5b6f6c04d88f-trusted-ca\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.885369 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.885340    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56566f0f-9187-433c-bc3f-5b6f6c04d88f-installation-pull-secrets\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.885472 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.885412    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56566f0f-9187-433c-bc3f-5b6f6c04d88f-bound-sa-token\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.885472 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.885440    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56566f0f-9187-433c-bc3f-5b6f6c04d88f-ca-trust-extracted\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.885575 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.885478    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56566f0f-9187-433c-bc3f-5b6f6c04d88f-image-registry-private-configuration\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.886203 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.885305    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56566f0f-9187-433c-bc3f-5b6f6c04d88f-registry-certificates\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.886203 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.884919    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56566f0f-9187-433c-bc3f-5b6f6c04d88f-trusted-ca\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.886659 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.886605    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56566f0f-9187-433c-bc3f-5b6f6c04d88f-ca-trust-extracted\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.886764 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.886698    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56566f0f-9187-433c-bc3f-5b6f6c04d88f-registry-tls\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.889005 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.888946    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56566f0f-9187-433c-bc3f-5b6f6c04d88f-installation-pull-secrets\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.889439 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.889316    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56566f0f-9187-433c-bc3f-5b6f6c04d88f-image-registry-private-configuration\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.946992 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.946254    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-944tk\" (UniqueName: \"kubernetes.io/projected/56566f0f-9187-433c-bc3f-5b6f6c04d88f-kube-api-access-944tk\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.956239 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.955754    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56566f0f-9187-433c-bc3f-5b6f6c04d88f-bound-sa-token\") pod \"image-registry-6c85cc4984-dfrbl\" (UID: \"56566f0f-9187-433c-bc3f-5b6f6c04d88f\") " pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:07.971594 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:07.971563    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:08.115836 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:08.115784    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7l7z" event={"ID":"c83cbbe5-3d68-4067-a0e9-c38f0f72df38","Type":"ContainerStarted","Data":"0613ee1a43e82b7b188b9ec563f4ee8395d06e6d825a26303c88913eaf6c39d8"}
Oct 31 00:29:08.222749 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:08.222705    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xf77w-4cxng"]
Oct 31 00:29:08.229304 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:29:08.229277    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e139718_2a1a_4b8f_b228_c8ceeda7fd70.slice/crio-64eb3c2007c82591ddf5584ae35d4bb89345f79a907abf4b7e6f9d223fa12014 WatchSource:0}: Error finding container 64eb3c2007c82591ddf5584ae35d4bb89345f79a907abf4b7e6f9d223fa12014: Status 404 returned error can't find the container with id 64eb3c2007c82591ddf5584ae35d4bb89345f79a907abf4b7e6f9d223fa12014
Oct 31 00:29:08.269497 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:08.267768    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qtfnw-lnbrj"]
Oct 31 00:29:08.271194 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:29:08.271165    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131dbd11_5c4f_4d96_81e9_86a13ab4ae0c.slice/crio-67b6851d5eb4a61751d8ab6ece61e1599b1d5d91513c01c834f2e02660579ca6 WatchSource:0}: Error finding container 67b6851d5eb4a61751d8ab6ece61e1599b1d5d91513c01c834f2e02660579ca6: Status 404 returned error can't find the container with id 67b6851d5eb4a61751d8ab6ece61e1599b1d5d91513c01c834f2e02660579ca6
Oct 31 00:29:08.274536 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:08.274510    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-c7h2h"]
Oct 31 00:29:08.359987 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:08.359948    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c85cc4984-dfrbl"]
Oct 31 00:29:09.124284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.123719    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qtfnw-lnbrj" event={"ID":"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c","Type":"ContainerStarted","Data":"c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5"}
Oct 31 00:29:09.124284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.123787    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qtfnw-lnbrj" event={"ID":"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c","Type":"ContainerStarted","Data":"67b6851d5eb4a61751d8ab6ece61e1599b1d5d91513c01c834f2e02660579ca6"}
Oct 31 00:29:09.126062 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.125984    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-c7h2h" event={"ID":"bc1d53d2-3d16-4d6f-97f5-e6c823857e4d","Type":"ContainerStarted","Data":"b1d9a0859f992692185f907470f2af1827d93d77b29d1b7dd4633defcdea84fb"}
Oct 31 00:29:09.127853 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.127779    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xf77w-4cxng" event={"ID":"7e139718-2a1a-4b8f-b228-c8ceeda7fd70","Type":"ContainerStarted","Data":"501d7cf73d0d3ff11687db56f48775157fde8eeaa86514685936dfff3954ad7b"}
Oct 31 00:29:09.127853 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.127824    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xf77w-4cxng" event={"ID":"7e139718-2a1a-4b8f-b228-c8ceeda7fd70","Type":"ContainerStarted","Data":"64eb3c2007c82591ddf5584ae35d4bb89345f79a907abf4b7e6f9d223fa12014"}
Oct 31 00:29:09.129539 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.129513    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl" event={"ID":"56566f0f-9187-433c-bc3f-5b6f6c04d88f","Type":"ContainerStarted","Data":"f23862a6544abd78d0092ed4a0c6edae89b068a91658feca61c0920d81d20a7c"}
Oct 31 00:29:09.129668 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.129557    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl" event={"ID":"56566f0f-9187-433c-bc3f-5b6f6c04d88f","Type":"ContainerStarted","Data":"b7079e2b45c1ad41c375319787829ddf16f9163559b6c8f4ed007699b35df3c9"}
Oct 31 00:29:09.130058 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.130018    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:09.132906 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.132883    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-r9hjn" event={"ID":"54ecc909-648d-4848-9f49-464285458129","Type":"ContainerStarted","Data":"2eb337baea3ed88a2ceb327090d74be2a41f743ffe83cf06a58e25364b303b97"}
Oct 31 00:29:09.235005 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.234943    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-qtfnw-lnbrj" podStartSLOduration=2.234926052 podStartE2EDuration="2.234926052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:09.23444498 +0000 UTC m=+21.416974277" watchObservedRunningTime="2026-10-31 00:29:09.234926052 +0000 UTC m=+21.417455339"
Oct 31 00:29:09.470056 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.465920    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl" podStartSLOduration=2.465900135 podStartE2EDuration="2.465900135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:09.35332127 +0000 UTC m=+21.535850565" watchObservedRunningTime="2026-10-31 00:29:09.465900135 +0000 UTC m=+21.648429430"
Oct 31 00:29:09.576936 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:09.576873    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-xf77w-4cxng" podStartSLOduration=2.576845338 podStartE2EDuration="2.576845338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:09.575662755 +0000 UTC m=+21.758192050" watchObservedRunningTime="2026-10-31 00:29:09.576845338 +0000 UTC m=+21.759374635"
Oct 31 00:29:10.145765 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:10.145731    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7l7z" event={"ID":"c83cbbe5-3d68-4067-a0e9-c38f0f72df38","Type":"ContainerStarted","Data":"ce7f031382932969fe612b3db9b8469df258ff99658f115b8d00223095e7dc28"}
Oct 31 00:29:10.434502 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:10.434471    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-grc8r_2a996538-9d06-447f-bf87-e7dbb8f078c6/dns/0.log"
Oct 31 00:29:10.544440 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:10.544414    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-grc8r_2a996538-9d06-447f-bf87-e7dbb8f078c6/kube-rbac-proxy/0.log"
Oct 31 00:29:10.649548 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:10.649521    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-29ctl_693a3c1f-8228-4691-bd99-f05fdf784cd4/dns-node-resolver/0.log"
Oct 31 00:29:10.866874 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:10.866842    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-grc8r_2a996538-9d06-447f-bf87-e7dbb8f078c6/dns/0.log"
Oct 31 00:29:10.966607 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:10.966551    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x"]
Oct 31 00:29:10.970638 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:10.970610    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x"
Oct 31 00:29:10.971957 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:10.971889    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-grc8r_2a996538-9d06-447f-bf87-e7dbb8f078c6/kube-rbac-proxy/0.log"
Oct 31 00:29:11.025143 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.025114    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x"]
Oct 31 00:29:11.026486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.026459    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c85cc4984-dfrbl_56566f0f-9187-433c-bc3f-5b6f6c04d88f/registry/0.log"
Oct 31 00:29:11.070284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.070258    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5k4xx\""
Oct 31 00:29:11.070431 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.070295    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 00:29:11.075817 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.075799    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-29ctl_693a3c1f-8228-4691-bd99-f05fdf784cd4/dns-node-resolver/0.log"
Oct 31 00:29:11.115327 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.115233    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/420c4e76-1007-4c7b-80b3-441ba4fdcf4a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7hw5x\" (UID: \"420c4e76-1007-4c7b-80b3-441ba4fdcf4a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x"
Oct 31 00:29:11.216918 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.216624    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/420c4e76-1007-4c7b-80b3-441ba4fdcf4a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7hw5x\" (UID: \"420c4e76-1007-4c7b-80b3-441ba4fdcf4a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x"
Oct 31 00:29:11.219398 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.219372    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/420c4e76-1007-4c7b-80b3-441ba4fdcf4a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-7hw5x\" (UID: \"420c4e76-1007-4c7b-80b3-441ba4fdcf4a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x"
Oct 31 00:29:11.222736 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.222710    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6cfc76855c-xm22c_76cd3516-aff9-4de2-a063-c30407f68564/registry/0.log"
Oct 31 00:29:11.284697 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.284663    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x"
Oct 31 00:29:11.423172 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.423082    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f66ccc5cb-h7cvd_d6511745-7315-463c-bc25-0e3e1fd6bf42/registry/0.log"
Oct 31 00:29:11.471201 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.471169    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c85cc4984-dfrbl_56566f0f-9187-433c-bc3f-5b6f6c04d88f/registry/0.log"
Oct 31 00:29:11.622884 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.622856    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fvksv_f5703958-5751-4c4d-b979-01585c84cd46/node-ca/0.log"
Oct 31 00:29:11.671264 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.671224    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6cfc76855c-xm22c_76cd3516-aff9-4de2-a063-c30407f68564/registry/0.log"
Oct 31 00:29:11.687596 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.687522    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x"]
Oct 31 00:29:11.694222 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:29:11.694195    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420c4e76_1007_4c7b_80b3_441ba4fdcf4a.slice/crio-120f2a188dd8d49ebbcd37cddfcbb670f91d1d41c90d975678701cf6f2fbf858 WatchSource:0}: Error finding container 120f2a188dd8d49ebbcd37cddfcbb670f91d1d41c90d975678701cf6f2fbf858: Status 404 returned error can't find the container with id 120f2a188dd8d49ebbcd37cddfcbb670f91d1d41c90d975678701cf6f2fbf858
Oct 31 00:29:11.871383 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:11.871351    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f66ccc5cb-h7cvd_d6511745-7315-463c-bc25-0e3e1fd6bf42/registry/0.log"
Oct 31 00:29:12.022897 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:12.022861    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55c864699c-n4cfn_2e209316-e440-4bdf-b2d9-cc79ac3e670f/router/0.log"
Oct 31 00:29:12.070950 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:12.070925    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fvksv_f5703958-5751-4c4d-b979-01585c84cd46/node-ca/0.log"
Oct 31 00:29:12.156008 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:12.155973    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x" event={"ID":"420c4e76-1007-4c7b-80b3-441ba4fdcf4a","Type":"ContainerStarted","Data":"120f2a188dd8d49ebbcd37cddfcbb670f91d1d41c90d975678701cf6f2fbf858"}
Oct 31 00:29:12.222693 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:12.222653    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmdnc_db65a944-e395-4d70-9807-c061aecf0584/serve-healthcheck-canary/0.log"
Oct 31 00:29:12.471990 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:12.471917    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55c864699c-n4cfn_2e209316-e440-4bdf-b2d9-cc79ac3e670f/router/0.log"
Oct 31 00:29:12.672009 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:12.671919    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jmdnc_db65a944-e395-4d70-9807-c061aecf0584/serve-healthcheck-canary/0.log"
Oct 31 00:29:13.394340 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.394302    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-86f6459c96-k7775"]
Oct 31 00:29:13.398166 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.398144    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.448306 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.448283    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 00:29:13.448471 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.448457    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 00:29:13.448526 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.448480    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 00:29:13.495269 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.495243    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 00:29:13.495881 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.495863    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 00:29:13.496006 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.495916    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tcw2z\""
Oct 31 00:29:13.496898 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.496879    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f6459c96-k7775"]
Oct 31 00:29:13.538403 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.538371    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-serving-cert\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.538403 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.538405    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-console-config\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.538611 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.538425    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-oauth-serving-cert\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.538611 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.538509    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hslt5\" (UniqueName: \"kubernetes.io/projected/e75e7499-0999-447c-9763-d0492dadbd6d-kube-api-access-hslt5\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.538611 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.538564    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-service-ca\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.538611 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.538597    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-oauth-config\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.538750 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.538672    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e75e7499-0999-447c-9763-d0492dadbd6d-tmp\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.639214 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.639178    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-oauth-config\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.639403 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.639227    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e75e7499-0999-447c-9763-d0492dadbd6d-tmp\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.639403 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.639375    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-serving-cert\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.639528 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.639415    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-console-config\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.639528 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.639441    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-oauth-serving-cert\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.639528 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.639487    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hslt5\" (UniqueName: \"kubernetes.io/projected/e75e7499-0999-447c-9763-d0492dadbd6d-kube-api-access-hslt5\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.639671 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.639539    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-service-ca\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.639671 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.639634    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e75e7499-0999-447c-9763-d0492dadbd6d-tmp\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.640269 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.640246    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-oauth-serving-cert\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.640385 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.640254    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-console-config\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.640482 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.640460    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-service-ca\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.642135 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.642114    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-oauth-config\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.642227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.642116    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-serving-cert\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.700803 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.700734    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hslt5\" (UniqueName: \"kubernetes.io/projected/e75e7499-0999-447c-9763-d0492dadbd6d-kube-api-access-hslt5\") pod \"console-86f6459c96-k7775\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") " pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:13.712436 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:13.712410    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:14.137714 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:14.137679    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86f6459c96-k7775"]
Oct 31 00:29:14.160291 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:29:14.160226    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:29:14.169963 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:14.169929    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x" event={"ID":"420c4e76-1007-4c7b-80b3-441ba4fdcf4a","Type":"ContainerStarted","Data":"62a80b5062465e8aacf74b698a1e0abd6c22230332a22d7e93e65d4a0e42a005"}
Oct 31 00:29:14.170166 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:14.170141    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x"
Oct 31 00:29:14.171326 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:14.171302    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f6459c96-k7775" event={"ID":"e75e7499-0999-447c-9763-d0492dadbd6d","Type":"ContainerStarted","Data":"a8eba3c4d4631868802c15120c2625d4ee9d4b30fc6f6bfcda58a651fbb9baed"}
Oct 31 00:29:14.176086 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:14.176062    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x"
Oct 31 00:29:14.283901 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:14.283836    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-7hw5x" podStartSLOduration=2.5369593999999998 podStartE2EDuration="4.283816225s" totalImagesPullingTime="1.746856825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:14.282460571 +0000 UTC m=+26.464989866" watchObservedRunningTime="2026-10-31 00:29:14.283816225 +0000 UTC m=+26.466345518"
Oct 31 00:29:14.725415 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:14.725386    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:29:15.775402 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:15.775368    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pdpbv"
Oct 31 00:29:16.507712 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:16.507684    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l64l2"
Oct 31 00:29:17.573094 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:17.572794    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:29:19.198684 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:19.198644    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f6459c96-k7775" event={"ID":"e75e7499-0999-447c-9763-d0492dadbd6d","Type":"ContainerStarted","Data":"b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee"}
Oct 31 00:29:19.312881 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:19.312825    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-86f6459c96-k7775" podStartSLOduration=2.134404307 podStartE2EDuration="6.312808524s" totalImagesPullingTime="4.178404217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:19.311836132 +0000 UTC m=+31.494365437" watchObservedRunningTime="2026-10-31 00:29:19.312808524 +0000 UTC m=+31.495337817"
Oct 31 00:29:23.241449 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.241415    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bn9lb"]
Oct 31 00:29:23.279582 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.279544    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.376639 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.376596    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 00:29:23.376639 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.376599    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 00:29:23.376892 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.376645    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mqw2c\""
Oct 31 00:29:23.376892 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.376652    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 00:29:23.376892 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.376645    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 00:29:23.432897 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.432853    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggdnz\" (UniqueName: \"kubernetes.io/projected/3318c24a-33bb-481b-8e63-0a4d5f032b93-kube-api-access-ggdnz\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.433081 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.432907    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3318c24a-33bb-481b-8e63-0a4d5f032b93-sys\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.433081 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.432935    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-accelerators-collector-config\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.433081 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.432997    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-textfile\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.433081 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.433019    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-tls\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.433081 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.433071    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.433335 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.433099    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-wtmp\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.433335 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.433116    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3318c24a-33bb-481b-8e63-0a4d5f032b93-metrics-client-ca\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.433335 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.433131    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3318c24a-33bb-481b-8e63-0a4d5f032b93-root\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534087 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.533992    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-textfile\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534087 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534029    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-tls\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534283 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534093    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534283 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534120    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-wtmp\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534283 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534139    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3318c24a-33bb-481b-8e63-0a4d5f032b93-metrics-client-ca\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534283 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534156    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3318c24a-33bb-481b-8e63-0a4d5f032b93-root\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534283 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534211    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggdnz\" (UniqueName: \"kubernetes.io/projected/3318c24a-33bb-481b-8e63-0a4d5f032b93-kube-api-access-ggdnz\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534283 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534239    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3318c24a-33bb-481b-8e63-0a4d5f032b93-sys\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534283 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534264    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-accelerators-collector-config\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534599 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534292    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-wtmp\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534599 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534353    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-textfile\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534599 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534356    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3318c24a-33bb-481b-8e63-0a4d5f032b93-sys\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534599 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534389    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3318c24a-33bb-481b-8e63-0a4d5f032b93-root\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534669    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3318c24a-33bb-481b-8e63-0a4d5f032b93-metrics-client-ca\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.534780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.534739    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-accelerators-collector-config\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.536751 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.536730    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.536751 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.536741    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3318c24a-33bb-481b-8e63-0a4d5f032b93-node-exporter-tls\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.600612 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.600578    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggdnz\" (UniqueName: \"kubernetes.io/projected/3318c24a-33bb-481b-8e63-0a4d5f032b93-kube-api-access-ggdnz\") pod \"node-exporter-bn9lb\" (UID: \"3318c24a-33bb-481b-8e63-0a4d5f032b93\") " pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.651813 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.651776    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bn9lb"
Oct 31 00:29:23.685324 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:29:23.685291    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3318c24a_33bb_481b_8e63_0a4d5f032b93.slice/crio-fa901b733816cfdbb3cc04c920a0aee4963caacf527838547c1132731187a7bb WatchSource:0}: Error finding container fa901b733816cfdbb3cc04c920a0aee4963caacf527838547c1132731187a7bb: Status 404 returned error can't find the container with id fa901b733816cfdbb3cc04c920a0aee4963caacf527838547c1132731187a7bb
Oct 31 00:29:23.713132 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.713098    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:23.713269 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.713170    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:23.719022 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:23.718994    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:24.227113 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:24.227072    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bn9lb" event={"ID":"3318c24a-33bb-481b-8e63-0a4d5f032b93","Type":"ContainerStarted","Data":"fa901b733816cfdbb3cc04c920a0aee4963caacf527838547c1132731187a7bb"}
Oct 31 00:29:24.231868 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:24.231842    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:26.239059 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:26.239012    4935 generic.go:356] "Generic (PLEG): container finished" podID="3318c24a-33bb-481b-8e63-0a4d5f032b93" containerID="6ee6656dcaf2908d8f3d3acf1c75a920b11480852a9180abd6ab0994cf15ad51" exitCode=0
Oct 31 00:29:26.239521 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:26.239106    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bn9lb" event={"ID":"3318c24a-33bb-481b-8e63-0a4d5f032b93","Type":"ContainerDied","Data":"6ee6656dcaf2908d8f3d3acf1c75a920b11480852a9180abd6ab0994cf15ad51"}
Oct 31 00:29:27.247280 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:27.247183    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bn9lb" event={"ID":"3318c24a-33bb-481b-8e63-0a4d5f032b93","Type":"ContainerStarted","Data":"e95d4184d137e0b18a741423aecf3ccdaa7feeb7934808f91cd5a2bf52426910"}
Oct 31 00:29:27.247280 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:27.247275    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bn9lb" event={"ID":"3318c24a-33bb-481b-8e63-0a4d5f032b93","Type":"ContainerStarted","Data":"f3fd9074e407545fdef7b12cf08d03ac190c0e4f71136dcbdf77ebe1dfaa45f2"}
Oct 31 00:29:27.393264 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:27.392010    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bn9lb" podStartSLOduration=2.611613247 podStartE2EDuration="4.391991298s" totalImagesPullingTime="1.696669097s" totalInitContainerRuntime="83.708954ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:27.362236097 +0000 UTC m=+39.544765395" watchObservedRunningTime="2026-10-31 00:29:27.391991298 +0000 UTC m=+39.574520593"
Oct 31 00:29:27.924224 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:27.924166    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f6459c96-k7775"]
Oct 31 00:29:31.155599 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:31.155566    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c85cc4984-dfrbl"
Oct 31 00:29:31.386473 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:31.386441    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cfc76855c-xm22c"]
Oct 31 00:29:32.589904 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.589848    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd" podUID="d6511745-7315-463c-bc25-0e3e1fd6bf42" containerName="registry" containerID="cri-o://9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7" gracePeriod=30
Oct 31 00:29:32.849289 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.849220    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:29:32.939019 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.938978    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-kube-api-access-9ppwn\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-kube-api-access-9ppwn\") pod \"d6511745-7315-463c-bc25-0e3e1fd6bf42\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") "
Oct 31 00:29:32.939204 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.939070    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-trusted-ca\") pod \"d6511745-7315-463c-bc25-0e3e1fd6bf42\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") "
Oct 31 00:29:32.939204 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.939109    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-image-registry-private-configuration\") pod \"d6511745-7315-463c-bc25-0e3e1fd6bf42\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") "
Oct 31 00:29:32.939204 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.939151    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6511745-7315-463c-bc25-0e3e1fd6bf42-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6511745-7315-463c-bc25-0e3e1fd6bf42-ca-trust-extracted\") pod \"d6511745-7315-463c-bc25-0e3e1fd6bf42\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") "
Oct 31 00:29:32.939353 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.939212    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-bound-sa-token\") pod \"d6511745-7315-463c-bc25-0e3e1fd6bf42\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") "
Oct 31 00:29:32.939353 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.939249    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-installation-pull-secrets\") pod \"d6511745-7315-463c-bc25-0e3e1fd6bf42\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") "
Oct 31 00:29:32.939353 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.939281    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") pod \"d6511745-7315-463c-bc25-0e3e1fd6bf42\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") "
Oct 31 00:29:32.939353 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.939329    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-certificates\") pod \"d6511745-7315-463c-bc25-0e3e1fd6bf42\" (UID: \"d6511745-7315-463c-bc25-0e3e1fd6bf42\") "
Oct 31 00:29:32.939611 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.939582    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-trusted-ca" pod "d6511745-7315-463c-bc25-0e3e1fd6bf42" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:32.939916 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.939891    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-certificates" pod "d6511745-7315-463c-bc25-0e3e1fd6bf42" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:32.942440 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.942410    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-kube-api-access-9ppwn" pod "d6511745-7315-463c-bc25-0e3e1fd6bf42" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42"). InnerVolumeSpecName "kube-api-access-9ppwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:32.942988 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.942963    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-installation-pull-secrets" pod "d6511745-7315-463c-bc25-0e3e1fd6bf42" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:32.943459 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.943425    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-bound-sa-token" pod "d6511745-7315-463c-bc25-0e3e1fd6bf42" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:32.944323 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.944288    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-image-registry-private-configuration" pod "d6511745-7315-463c-bc25-0e3e1fd6bf42" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:32.944840 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.944813    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls" pod "d6511745-7315-463c-bc25-0e3e1fd6bf42" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:32.948160 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:32.948116    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6511745-7315-463c-bc25-0e3e1fd6bf42-ca-trust-extracted" pod "d6511745-7315-463c-bc25-0e3e1fd6bf42" (UID: "d6511745-7315-463c-bc25-0e3e1fd6bf42"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:33.040767 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.040729    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ppwn\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-kube-api-access-9ppwn\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.040767 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.040769    4935 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-trusted-ca\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.041019 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.040783    4935 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-image-registry-private-configuration\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.041019 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.040794    4935 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d6511745-7315-463c-bc25-0e3e1fd6bf42-ca-trust-extracted\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.041019 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.040807    4935 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-bound-sa-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.041019 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.040817    4935 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d6511745-7315-463c-bc25-0e3e1fd6bf42-installation-pull-secrets\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.041019 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.040826    4935 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-tls\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.041019 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.040840    4935 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d6511745-7315-463c-bc25-0e3e1fd6bf42-registry-certificates\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:33.282999 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.282962    4935 generic.go:356] "Generic (PLEG): container finished" podID="d6511745-7315-463c-bc25-0e3e1fd6bf42" containerID="9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7" exitCode=0
Oct 31 00:29:33.283181 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.283008    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"
Oct 31 00:29:33.283181 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.283064    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd" event={"ID":"d6511745-7315-463c-bc25-0e3e1fd6bf42","Type":"ContainerDied","Data":"9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7"}
Oct 31 00:29:33.283181 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.283121    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f66ccc5cb-h7cvd" event={"ID":"d6511745-7315-463c-bc25-0e3e1fd6bf42","Type":"ContainerDied","Data":"5d7d9e9ed4b036e947f1ccddc9156f3ab071a089d925f1e1c3782aa047ff4c32"}
Oct 31 00:29:33.283181 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.283139    4935 scope.go:122] "RemoveContainer" containerID="9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7"
Oct 31 00:29:33.285382 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.285356    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-c7h2h" event={"ID":"bc1d53d2-3d16-4d6f-97f5-e6c823857e4d","Type":"ContainerStarted","Data":"1db8a875cc224686d952ad91cc6d3e1685b4b6c8298afa10d98e01ddb2f795b1"}
Oct 31 00:29:33.285642 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.285570    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-c7h2h"
Oct 31 00:29:33.287315 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.287294    4935 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-c7h2h container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body=
Oct 31 00:29:33.287421 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.287336    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-c7h2h" podUID="bc1d53d2-3d16-4d6f-97f5-e6c823857e4d" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused"
Oct 31 00:29:33.292244 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.292216    4935 scope.go:122] "RemoveContainer" containerID="9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7"
Oct 31 00:29:33.292594 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:29:33.292568    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7\": container with ID starting with 9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7 not found: ID does not exist" containerID="9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7"
Oct 31 00:29:33.292676 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.292609    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7"} err="failed to get container status \"9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7\": rpc error: code = NotFound desc = could not find container \"9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7\": container with ID starting with 9a43e8b01b8ed4b9ae2bff825acdf318b72573070f095a7a8f2a52ceb37c5ec7 not found: ID does not exist"
Oct 31 00:29:33.449330 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.449251    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"]
Oct 31 00:29:33.456116 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.455901    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f66ccc5cb-h7cvd"]
Oct 31 00:29:33.945460 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:33.945417    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6511745-7315-463c-bc25-0e3e1fd6bf42" path="/var/lib/kubelet/pods/d6511745-7315-463c-bc25-0e3e1fd6bf42/volumes"
Oct 31 00:29:34.312990 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:34.312957    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-c7h2h"
Oct 31 00:29:34.520601 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:34.520546    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-c7h2h" podStartSLOduration=2.6362856150000002 podStartE2EDuration="27.520530807s" totalImagesPullingTime="24.884245192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:33.570371266 +0000 UTC m=+45.752900561" watchObservedRunningTime="2026-10-31 00:29:34.520530807 +0000 UTC m=+46.703060105"
Oct 31 00:29:44.152258 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:29:44.152216    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:29:52.951539 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:52.951499    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-86f6459c96-k7775" podUID="e75e7499-0999-447c-9763-d0492dadbd6d" containerName="console" containerID="cri-o://b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee" gracePeriod=15
Oct 31 00:29:53.266563 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.266528    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:53.317166 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317139    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-console-config\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-console-config\") pod \"e75e7499-0999-447c-9763-d0492dadbd6d\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") "
Oct 31 00:29:53.317295 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317171    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-oauth-config\") pod \"e75e7499-0999-447c-9763-d0492dadbd6d\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") "
Oct 31 00:29:53.317295 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317207    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e75e7499-0999-447c-9763-d0492dadbd6d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e75e7499-0999-447c-9763-d0492dadbd6d-tmp\") pod \"e75e7499-0999-447c-9763-d0492dadbd6d\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") "
Oct 31 00:29:53.317295 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317248    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-serving-cert\") pod \"e75e7499-0999-447c-9763-d0492dadbd6d\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") "
Oct 31 00:29:53.317295 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317282    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75e7499-0999-447c-9763-d0492dadbd6d-kube-api-access-hslt5\" (UniqueName: \"kubernetes.io/projected/e75e7499-0999-447c-9763-d0492dadbd6d-kube-api-access-hslt5\") pod \"e75e7499-0999-447c-9763-d0492dadbd6d\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") "
Oct 31 00:29:53.317485 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317345    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-oauth-serving-cert\") pod \"e75e7499-0999-447c-9763-d0492dadbd6d\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") "
Oct 31 00:29:53.317485 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317413    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-service-ca\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-service-ca\") pod \"e75e7499-0999-447c-9763-d0492dadbd6d\" (UID: \"e75e7499-0999-447c-9763-d0492dadbd6d\") "
Oct 31 00:29:53.317588 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317550    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e75e7499-0999-447c-9763-d0492dadbd6d-tmp" pod "e75e7499-0999-447c-9763-d0492dadbd6d" (UID: "e75e7499-0999-447c-9763-d0492dadbd6d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:53.317644 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317587    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-console-config" pod "e75e7499-0999-447c-9763-d0492dadbd6d" (UID: "e75e7499-0999-447c-9763-d0492dadbd6d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:53.317774 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317696    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-oauth-serving-cert" pod "e75e7499-0999-447c-9763-d0492dadbd6d" (UID: "e75e7499-0999-447c-9763-d0492dadbd6d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:53.317860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317704    4935 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-console-config\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:53.317860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317817    4935 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e75e7499-0999-447c-9763-d0492dadbd6d-tmp\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:53.317860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.317849    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-service-ca" pod "e75e7499-0999-447c-9763-d0492dadbd6d" (UID: "e75e7499-0999-447c-9763-d0492dadbd6d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:53.319575 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.319549    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-serving-cert" pod "e75e7499-0999-447c-9763-d0492dadbd6d" (UID: "e75e7499-0999-447c-9763-d0492dadbd6d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:53.319670 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.319605    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75e7499-0999-447c-9763-d0492dadbd6d-kube-api-access-hslt5" pod "e75e7499-0999-447c-9763-d0492dadbd6d" (UID: "e75e7499-0999-447c-9763-d0492dadbd6d"). InnerVolumeSpecName "kube-api-access-hslt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:53.319670 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.319630    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-oauth-config" pod "e75e7499-0999-447c-9763-d0492dadbd6d" (UID: "e75e7499-0999-447c-9763-d0492dadbd6d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:53.375408 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.375376    4935 generic.go:356] "Generic (PLEG): container finished" podID="e75e7499-0999-447c-9763-d0492dadbd6d" containerID="b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee" exitCode=0
Oct 31 00:29:53.375544 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.375422    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86f6459c96-k7775"
Oct 31 00:29:53.375544 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.375467    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f6459c96-k7775" event={"ID":"e75e7499-0999-447c-9763-d0492dadbd6d","Type":"ContainerDied","Data":"b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee"}
Oct 31 00:29:53.375544 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.375523    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86f6459c96-k7775" event={"ID":"e75e7499-0999-447c-9763-d0492dadbd6d","Type":"ContainerDied","Data":"a8eba3c4d4631868802c15120c2625d4ee9d4b30fc6f6bfcda58a651fbb9baed"}
Oct 31 00:29:53.375717 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.375549    4935 scope.go:122] "RemoveContainer" containerID="b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee"
Oct 31 00:29:53.385197 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.385167    4935 scope.go:122] "RemoveContainer" containerID="b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee"
Oct 31 00:29:53.385619 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:29:53.385572    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee\": container with ID starting with b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee not found: ID does not exist" containerID="b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee"
Oct 31 00:29:53.385716 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.385622    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee"} err="failed to get container status \"b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee\": rpc error: code = NotFound desc = could not find container \"b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee\": container with ID starting with b51a694afd51a29abad979199ae27ebafdeeef0d8a559add038a864aed9e8bee not found: ID does not exist"
Oct 31 00:29:53.419157 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.419135    4935 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-serving-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:53.419157 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.419157    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hslt5\" (UniqueName: \"kubernetes.io/projected/e75e7499-0999-447c-9763-d0492dadbd6d-kube-api-access-hslt5\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:53.419309 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.419169    4935 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-oauth-serving-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:53.419309 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.419178    4935 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e75e7499-0999-447c-9763-d0492dadbd6d-service-ca\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:53.419309 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.419187    4935 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e75e7499-0999-447c-9763-d0492dadbd6d-console-oauth-config\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:53.541026 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.540990    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86f6459c96-k7775"]
Oct 31 00:29:53.548984 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.548949    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86f6459c96-k7775"]
Oct 31 00:29:53.943009 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:53.942933    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e75e7499-0999-447c-9763-d0492dadbd6d" path="/var/lib/kubelet/pods/e75e7499-0999-447c-9763-d0492dadbd6d/volumes"
Oct 31 00:29:55.386228 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:55.386197    4935 generic.go:356] "Generic (PLEG): container finished" podID="f59717aa-c796-4074-b63f-2fc7e13f3633" containerID="f46f42218a8fac61fc4001c0fec2d5ebd23dff78b8768abea378ffff83a072d1" exitCode=0
Oct 31 00:29:55.386701 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:55.386275    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng" event={"ID":"f59717aa-c796-4074-b63f-2fc7e13f3633","Type":"ContainerDied","Data":"f46f42218a8fac61fc4001c0fec2d5ebd23dff78b8768abea378ffff83a072d1"}
Oct 31 00:29:55.386701 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:55.386667    4935 scope.go:122] "RemoveContainer" containerID="f46f42218a8fac61fc4001c0fec2d5ebd23dff78b8768abea378ffff83a072d1"
Oct 31 00:29:56.392315 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.392274    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ldsng" event={"ID":"f59717aa-c796-4074-b63f-2fc7e13f3633","Type":"ContainerStarted","Data":"220d201799c9cd23520986654c27704620ea8503fcbfc7e18bc1cdbfc38c7aec"}
Oct 31 00:29:56.412468 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.412435    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c" podUID="76cd3516-aff9-4de2-a063-c30407f68564" containerName="registry" containerID="cri-o://aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc" gracePeriod=30
Oct 31 00:29:56.682254 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.682229    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:29:56.748790 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.748757    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-kube-api-access-xfmz7\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-kube-api-access-xfmz7\") pod \"76cd3516-aff9-4de2-a063-c30407f68564\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") "
Oct 31 00:29:56.748957 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.748796    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-installation-pull-secrets\") pod \"76cd3516-aff9-4de2-a063-c30407f68564\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") "
Oct 31 00:29:56.748957 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.748832    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-bound-sa-token\") pod \"76cd3516-aff9-4de2-a063-c30407f68564\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") "
Oct 31 00:29:56.748957 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.748919    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76cd3516-aff9-4de2-a063-c30407f68564-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76cd3516-aff9-4de2-a063-c30407f68564-ca-trust-extracted\") pod \"76cd3516-aff9-4de2-a063-c30407f68564\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") "
Oct 31 00:29:56.749122 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.748967    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") pod \"76cd3516-aff9-4de2-a063-c30407f68564\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") "
Oct 31 00:29:56.749122 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.749025    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-registry-certificates\") pod \"76cd3516-aff9-4de2-a063-c30407f68564\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") "
Oct 31 00:29:56.749122 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.749090    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-trusted-ca\") pod \"76cd3516-aff9-4de2-a063-c30407f68564\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") "
Oct 31 00:29:56.749809 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.749317    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-image-registry-private-configuration\") pod \"76cd3516-aff9-4de2-a063-c30407f68564\" (UID: \"76cd3516-aff9-4de2-a063-c30407f68564\") "
Oct 31 00:29:56.749809 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.749505    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-trusted-ca" pod "76cd3516-aff9-4de2-a063-c30407f68564" (UID: "76cd3516-aff9-4de2-a063-c30407f68564"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:56.749809 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.749520    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-registry-certificates" pod "76cd3516-aff9-4de2-a063-c30407f68564" (UID: "76cd3516-aff9-4de2-a063-c30407f68564"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:56.749809 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.749800    4935 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-registry-certificates\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.750259 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.749820    4935 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76cd3516-aff9-4de2-a063-c30407f68564-trusted-ca\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.751402 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.751373    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-kube-api-access-xfmz7" pod "76cd3516-aff9-4de2-a063-c30407f68564" (UID: "76cd3516-aff9-4de2-a063-c30407f68564"). InnerVolumeSpecName "kube-api-access-xfmz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:56.751504 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.751426    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-bound-sa-token" pod "76cd3516-aff9-4de2-a063-c30407f68564" (UID: "76cd3516-aff9-4de2-a063-c30407f68564"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:56.751630 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.751598    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-installation-pull-secrets" pod "76cd3516-aff9-4de2-a063-c30407f68564" (UID: "76cd3516-aff9-4de2-a063-c30407f68564"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:56.751965 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.751940    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls" pod "76cd3516-aff9-4de2-a063-c30407f68564" (UID: "76cd3516-aff9-4de2-a063-c30407f68564"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:56.752450 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.752430    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-image-registry-private-configuration" pod "76cd3516-aff9-4de2-a063-c30407f68564" (UID: "76cd3516-aff9-4de2-a063-c30407f68564"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:56.756429 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.756327    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cd3516-aff9-4de2-a063-c30407f68564-ca-trust-extracted" pod "76cd3516-aff9-4de2-a063-c30407f68564" (UID: "76cd3516-aff9-4de2-a063-c30407f68564"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:29:56.851005 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.850978    4935 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-image-registry-private-configuration\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.851005 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.851004    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfmz7\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-kube-api-access-xfmz7\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.851198 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.851017    4935 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76cd3516-aff9-4de2-a063-c30407f68564-installation-pull-secrets\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.851198 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.851027    4935 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-bound-sa-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.851198 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.851057    4935 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76cd3516-aff9-4de2-a063-c30407f68564-ca-trust-extracted\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:56.851198 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:56.851067    4935 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76cd3516-aff9-4de2-a063-c30407f68564-registry-tls\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:57.400887 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:57.400850    4935 generic.go:356] "Generic (PLEG): container finished" podID="76cd3516-aff9-4de2-a063-c30407f68564" containerID="aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc" exitCode=0
Oct 31 00:29:57.401357 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:57.400897    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c"
Oct 31 00:29:57.401357 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:57.400933    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c" event={"ID":"76cd3516-aff9-4de2-a063-c30407f68564","Type":"ContainerDied","Data":"aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc"}
Oct 31 00:29:57.401357 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:57.400994    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cfc76855c-xm22c" event={"ID":"76cd3516-aff9-4de2-a063-c30407f68564","Type":"ContainerDied","Data":"d4dbc08c75fc4b8c82f55fc48380d0fc57837ed28e2c1d919531492e692b2a3d"}
Oct 31 00:29:57.401357 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:57.401016    4935 scope.go:122] "RemoveContainer" containerID="aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc"
Oct 31 00:29:57.409873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:57.409858    4935 scope.go:122] "RemoveContainer" containerID="aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc"
Oct 31 00:29:57.410186 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:29:57.410168    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc\": container with ID starting with aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc not found: ID does not exist" containerID="aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc"
Oct 31 00:29:57.410232 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:57.410197    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc"} err="failed to get container status \"aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc\": rpc error: code = NotFound desc = could not find container \"aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc\": container with ID starting with aa3f74f083e44c7730f2f67f6fd86d7888d1446f0a7280149dfa16baf257a5dc not found: ID does not exist"
Oct 31 00:29:57.562815 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:57.562779    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cfc76855c-xm22c"]
Oct 31 00:29:57.609467 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:57.609433    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cfc76855c-xm22c"]
Oct 31 00:29:57.947020 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:29:57.946985    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76cd3516-aff9-4de2-a063-c30407f68564" path="/var/lib/kubelet/pods/76cd3516-aff9-4de2-a063-c30407f68564/volumes"
Oct 31 00:30:14.152164 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:30:14.152124    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:30:14.476300 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:14.476219    4935 generic.go:356] "Generic (PLEG): container finished" podID="990975fd-2466-4224-acbe-6269e49a9ef9" containerID="adcd0085e6dc85733c698f35b6382f36d370640819e168a7e67235cad3370832" exitCode=0
Oct 31 00:30:14.476300 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:14.476266    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8" event={"ID":"990975fd-2466-4224-acbe-6269e49a9ef9","Type":"ContainerDied","Data":"adcd0085e6dc85733c698f35b6382f36d370640819e168a7e67235cad3370832"}
Oct 31 00:30:14.476601 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:14.476586    4935 scope.go:122] "RemoveContainer" containerID="adcd0085e6dc85733c698f35b6382f36d370640819e168a7e67235cad3370832"
Oct 31 00:30:15.483359 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:15.483325    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-t7fd8" event={"ID":"990975fd-2466-4224-acbe-6269e49a9ef9","Type":"ContainerStarted","Data":"24b2a5fdcbe2c5ccf86879df794a750dee73c258852181ebd6de4a39cf18f626"}
Oct 31 00:30:16.489651 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:16.489614    4935 generic.go:356] "Generic (PLEG): container finished" podID="f53b8d98-bbb7-4108-a65d-6a15ab782bee" containerID="8ab125f20dccff9382c944f4b13b771a64ca6b8bd5e251fa3b4ce1bd10024a56" exitCode=0
Oct 31 00:30:16.490067 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:16.489687    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4sch4" event={"ID":"f53b8d98-bbb7-4108-a65d-6a15ab782bee","Type":"ContainerDied","Data":"8ab125f20dccff9382c944f4b13b771a64ca6b8bd5e251fa3b4ce1bd10024a56"}
Oct 31 00:30:16.490067 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:16.489769    4935 scope.go:122] "RemoveContainer" containerID="b81900eec2c2405e13a75b226b2fd7840b8cabb06ac2c5be5d7e034aac4b28e6"
Oct 31 00:30:16.490198 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:16.490112    4935 scope.go:122] "RemoveContainer" containerID="8ab125f20dccff9382c944f4b13b771a64ca6b8bd5e251fa3b4ce1bd10024a56"
Oct 31 00:30:16.490343 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:30:16.490324    4935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-4sch4_openshift-insights(f53b8d98-bbb7-4108-a65d-6a15ab782bee)\"" pod="openshift-insights/insights-operator-7c4986d449-4sch4" podUID="f53b8d98-bbb7-4108-a65d-6a15ab782bee"
Oct 31 00:30:22.636208 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.636173    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d4d64fb86-hvmvn"]
Oct 31 00:30:22.636700 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.636512    4935 cpu_manager.go:414] "containerMap: removing container" podUID="76cd3516-aff9-4de2-a063-c30407f68564" containerName="registry"
Oct 31 00:30:22.636700 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.636523    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76cd3516-aff9-4de2-a063-c30407f68564" containerName="registry"
Oct 31 00:30:22.636700 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.636533    4935 cpu_manager.go:414] "containerMap: removing container" podUID="d6511745-7315-463c-bc25-0e3e1fd6bf42" containerName="registry"
Oct 31 00:30:22.636700 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.636538    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6511745-7315-463c-bc25-0e3e1fd6bf42" containerName="registry"
Oct 31 00:30:22.636700 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.636553    4935 cpu_manager.go:414] "containerMap: removing container" podUID="e75e7499-0999-447c-9763-d0492dadbd6d" containerName="console"
Oct 31 00:30:22.636700 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.636557    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e75e7499-0999-447c-9763-d0492dadbd6d" containerName="console"
Oct 31 00:30:22.636700 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.636612    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6511745-7315-463c-bc25-0e3e1fd6bf42" containerName="registry"
Oct 31 00:30:22.636700 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.636621    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e75e7499-0999-447c-9763-d0492dadbd6d" containerName="console"
Oct 31 00:30:22.636700 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.636631    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="76cd3516-aff9-4de2-a063-c30407f68564" containerName="registry"
Oct 31 00:30:22.639173 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.639153    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.669746 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.669717    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b4d5e98-088b-4389-a49c-7a5dd7890cee-console-serving-cert\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.669746 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.669746    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-oauth-serving-cert\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.669899 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.669773    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-console-config\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.669899 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.669790    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-trusted-ca-bundle\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.669962 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.669915    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxmq9\" (UniqueName: \"kubernetes.io/projected/0b4d5e98-088b-4389-a49c-7a5dd7890cee-kube-api-access-pxmq9\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.669962 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.669948    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b4d5e98-088b-4389-a49c-7a5dd7890cee-console-oauth-config\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.670084 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.669963    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-service-ca\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.670084 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.670055    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b4d5e98-088b-4389-a49c-7a5dd7890cee-tmp\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.737526 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.737462    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 00:30:22.737526 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.737507    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 00:30:22.737526 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.737467    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 00:30:22.737811 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.737562    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 00:30:22.737811 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.737591    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 00:30:22.737811 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.737635    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tcw2z\""
Oct 31 00:30:22.770640 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.770609    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b4d5e98-088b-4389-a49c-7a5dd7890cee-console-oauth-config\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.770640 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.770639    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-service-ca\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.770855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.770677    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b4d5e98-088b-4389-a49c-7a5dd7890cee-tmp\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.770855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.770720    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b4d5e98-088b-4389-a49c-7a5dd7890cee-console-serving-cert\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.770855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.770742    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-oauth-serving-cert\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.771007 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.770885    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-console-config\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.771007 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.770932    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-trusted-ca-bundle\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.771007 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.770980    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxmq9\" (UniqueName: \"kubernetes.io/projected/0b4d5e98-088b-4389-a49c-7a5dd7890cee-kube-api-access-pxmq9\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.771232 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.771213    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b4d5e98-088b-4389-a49c-7a5dd7890cee-tmp\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.771503 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.771448    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-service-ca\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.771686 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.771562    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-console-config\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.771797 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.771769    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-oauth-serving-cert\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.773308 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.773286    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b4d5e98-088b-4389-a49c-7a5dd7890cee-console-oauth-config\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.773401 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.773361    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b4d5e98-088b-4389-a49c-7a5dd7890cee-console-serving-cert\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.832309 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.832271    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 00:30:22.833478 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.833454    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4d64fb86-hvmvn"]
Oct 31 00:30:22.843542 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.843517    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4d5e98-088b-4389-a49c-7a5dd7890cee-trusted-ca-bundle\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.874774 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.874750    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxmq9\" (UniqueName: \"kubernetes.io/projected/0b4d5e98-088b-4389-a49c-7a5dd7890cee-kube-api-access-pxmq9\") pod \"console-7d4d64fb86-hvmvn\" (UID: \"0b4d5e98-088b-4389-a49c-7a5dd7890cee\") " pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:22.948854 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:22.948771    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:23.319936 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:23.319900    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4d64fb86-hvmvn"]
Oct 31 00:30:23.326013 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:30:23.325985    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4d5e98_088b_4389_a49c_7a5dd7890cee.slice/crio-feeea8e31d3f6722cdb343e0ae7f72f4f784f16d6e0a5f406cca2d28e579c7d2 WatchSource:0}: Error finding container feeea8e31d3f6722cdb343e0ae7f72f4f784f16d6e0a5f406cca2d28e579c7d2: Status 404 returned error can't find the container with id feeea8e31d3f6722cdb343e0ae7f72f4f784f16d6e0a5f406cca2d28e579c7d2
Oct 31 00:30:23.519438 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:23.519398    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4d64fb86-hvmvn" event={"ID":"0b4d5e98-088b-4389-a49c-7a5dd7890cee","Type":"ContainerStarted","Data":"ce167d4b1c4b9c800f764e07fbd66caa6103c9c0af4ce3d9725e33a2c8b55ba6"}
Oct 31 00:30:23.519621 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:23.519459    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4d64fb86-hvmvn" event={"ID":"0b4d5e98-088b-4389-a49c-7a5dd7890cee","Type":"ContainerStarted","Data":"feeea8e31d3f6722cdb343e0ae7f72f4f784f16d6e0a5f406cca2d28e579c7d2"}
Oct 31 00:30:23.634501 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:23.634401    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7d4d64fb86-hvmvn" podStartSLOduration=1.634383725 podStartE2EDuration="1.634383725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:23.633245682 +0000 UTC m=+95.815774976" watchObservedRunningTime="2026-10-31 00:30:23.634383725 +0000 UTC m=+95.816913131"
Oct 31 00:30:28.937963 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:28.937929    4935 scope.go:122] "RemoveContainer" containerID="8ab125f20dccff9382c944f4b13b771a64ca6b8bd5e251fa3b4ce1bd10024a56"
Oct 31 00:30:29.550093 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:29.550053    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4sch4" event={"ID":"f53b8d98-bbb7-4108-a65d-6a15ab782bee","Type":"ContainerStarted","Data":"3d19212eb1a19f38b6c35f8e43abeab8cd8def59dbd50241db0e048caf237e12"}
Oct 31 00:30:30.155373 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.155335    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-qtfnw-lnbrj"]
Oct 31 00:30:30.155789 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.155526    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-qtfnw-lnbrj" podUID="131dbd11-5c4f-4d96-81e9-86a13ab4ae0c" containerName="insights-gathering" containerID="cri-o://c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5" gracePeriod=30
Oct 31 00:30:30.351446 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.351414    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-xf77w-4cxng"]
Oct 31 00:30:30.351639 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.351619    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-xf77w-4cxng" podUID="7e139718-2a1a-4b8f-b228-c8ceeda7fd70" containerName="insights-gathering" containerID="cri-o://501d7cf73d0d3ff11687db56f48775157fde8eeaa86514685936dfff3954ad7b" gracePeriod=30
Oct 31 00:30:30.475841 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.475810    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:30:30.537363 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.537327    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-service-ca-bundle\") pod \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") "
Oct 31 00:30:30.537543 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.537394    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-kube-api-access-6kfj6\" (UniqueName: \"kubernetes.io/projected/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-kube-api-access-6kfj6\") pod \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") "
Oct 31 00:30:30.537543 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.537430    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-archives-path\") pod \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") "
Oct 31 00:30:30.537650 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.537567    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-trusted-ca-bundle\") pod \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\" (UID: \"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c\") "
Oct 31 00:30:30.538106 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.537758    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-service-ca-bundle" pod "131dbd11-5c4f-4d96-81e9-86a13ab4ae0c" (UID: "131dbd11-5c4f-4d96-81e9-86a13ab4ae0c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:30.538106 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.538058    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-archives-path" pod "131dbd11-5c4f-4d96-81e9-86a13ab4ae0c" (UID: "131dbd11-5c4f-4d96-81e9-86a13ab4ae0c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:30.538106 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.538082    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-trusted-ca-bundle" pod "131dbd11-5c4f-4d96-81e9-86a13ab4ae0c" (UID: "131dbd11-5c4f-4d96-81e9-86a13ab4ae0c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:30.540229 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.540207    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-kube-api-access-6kfj6" pod "131dbd11-5c4f-4d96-81e9-86a13ab4ae0c" (UID: "131dbd11-5c4f-4d96-81e9-86a13ab4ae0c"). InnerVolumeSpecName "kube-api-access-6kfj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:30.557643 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.557610    4935 generic.go:356] "Generic (PLEG): container finished" podID="131dbd11-5c4f-4d96-81e9-86a13ab4ae0c" containerID="c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5" exitCode=2
Oct 31 00:30:30.557800 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.557675    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qtfnw-lnbrj"
Oct 31 00:30:30.557800 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.557701    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qtfnw-lnbrj" event={"ID":"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c","Type":"ContainerDied","Data":"c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5"}
Oct 31 00:30:30.557800 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.557761    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qtfnw-lnbrj" event={"ID":"131dbd11-5c4f-4d96-81e9-86a13ab4ae0c","Type":"ContainerDied","Data":"67b6851d5eb4a61751d8ab6ece61e1599b1d5d91513c01c834f2e02660579ca6"}
Oct 31 00:30:30.557800 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.557786    4935 scope.go:122] "RemoveContainer" containerID="c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5"
Oct 31 00:30:30.559402 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.559381    4935 generic.go:356] "Generic (PLEG): container finished" podID="7e139718-2a1a-4b8f-b228-c8ceeda7fd70" containerID="501d7cf73d0d3ff11687db56f48775157fde8eeaa86514685936dfff3954ad7b" exitCode=2
Oct 31 00:30:30.559528 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.559452    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xf77w-4cxng" event={"ID":"7e139718-2a1a-4b8f-b228-c8ceeda7fd70","Type":"ContainerDied","Data":"501d7cf73d0d3ff11687db56f48775157fde8eeaa86514685936dfff3954ad7b"}
Oct 31 00:30:30.566881 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.566862    4935 scope.go:122] "RemoveContainer" containerID="c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5"
Oct 31 00:30:30.567364 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:30:30.567330    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5\": container with ID starting with c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5 not found: ID does not exist" containerID="c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5"
Oct 31 00:30:30.567454 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.567392    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5"} err="failed to get container status \"c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5\": rpc error: code = NotFound desc = could not find container \"c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5\": container with ID starting with c3019e8cbf3dfced406ee91d6939c186c9e334bd7fc83b15a9430c797ddd24c5 not found: ID does not exist"
Oct 31 00:30:30.612621 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.612599    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:30:30.638364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.638335    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-archives-path\") pod \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") "
Oct 31 00:30:30.638528 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.638383    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-service-ca-bundle\") pod \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") "
Oct 31 00:30:30.638528 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.638405    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-trusted-ca-bundle\") pod \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") "
Oct 31 00:30:30.638528 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.638429    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-kube-api-access-q6tc5\" (UniqueName: \"kubernetes.io/projected/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-kube-api-access-q6tc5\") pod \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\" (UID: \"7e139718-2a1a-4b8f-b228-c8ceeda7fd70\") "
Oct 31 00:30:30.638692 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.638601    4935 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-trusted-ca-bundle\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.638692 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.638619    4935 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-service-ca-bundle\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.638692 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.638633    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kfj6\" (UniqueName: \"kubernetes.io/projected/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-kube-api-access-6kfj6\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.638692 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.638652    4935 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c-archives-path\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.638899 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.638665    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-archives-path" pod "7e139718-2a1a-4b8f-b228-c8ceeda7fd70" (UID: "7e139718-2a1a-4b8f-b228-c8ceeda7fd70"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:30.638899 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.638750    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-service-ca-bundle" pod "7e139718-2a1a-4b8f-b228-c8ceeda7fd70" (UID: "7e139718-2a1a-4b8f-b228-c8ceeda7fd70"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:30.638899 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.638816    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-trusted-ca-bundle" pod "7e139718-2a1a-4b8f-b228-c8ceeda7fd70" (UID: "7e139718-2a1a-4b8f-b228-c8ceeda7fd70"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:30.640696 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.640673    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-kube-api-access-q6tc5" pod "7e139718-2a1a-4b8f-b228-c8ceeda7fd70" (UID: "7e139718-2a1a-4b8f-b228-c8ceeda7fd70"). InnerVolumeSpecName "kube-api-access-q6tc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:30.695483 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.695403    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-qtfnw-lnbrj"]
Oct 31 00:30:30.701760 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.701734    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-qtfnw-lnbrj"]
Oct 31 00:30:30.739369 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.739340    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6tc5\" (UniqueName: \"kubernetes.io/projected/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-kube-api-access-q6tc5\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.739369 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.739364    4935 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-archives-path\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.739549 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.739376    4935 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-service-ca-bundle\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.739549 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:30.739387    4935 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e139718-2a1a-4b8f-b228-c8ceeda7fd70-trusted-ca-bundle\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:31.564915 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:31.564885    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xf77w-4cxng"
Oct 31 00:30:31.565451 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:31.564917    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xf77w-4cxng" event={"ID":"7e139718-2a1a-4b8f-b228-c8ceeda7fd70","Type":"ContainerDied","Data":"64eb3c2007c82591ddf5584ae35d4bb89345f79a907abf4b7e6f9d223fa12014"}
Oct 31 00:30:31.565451 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:31.564976    4935 scope.go:122] "RemoveContainer" containerID="501d7cf73d0d3ff11687db56f48775157fde8eeaa86514685936dfff3954ad7b"
Oct 31 00:30:31.722531 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:31.722498    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-xf77w-4cxng"]
Oct 31 00:30:31.768969 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:31.768925    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-xf77w-4cxng"]
Oct 31 00:30:31.942890 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:31.942798    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131dbd11-5c4f-4d96-81e9-86a13ab4ae0c" path="/var/lib/kubelet/pods/131dbd11-5c4f-4d96-81e9-86a13ab4ae0c/volumes"
Oct 31 00:30:31.943281 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:31.943261    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e139718-2a1a-4b8f-b228-c8ceeda7fd70" path="/var/lib/kubelet/pods/7e139718-2a1a-4b8f-b228-c8ceeda7fd70/volumes"
Oct 31 00:30:32.949599 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:32.949568    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:32.950060 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:32.949686    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:32.954422 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:32.954395    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:33.578164 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:30:33.578137    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d4d64fb86-hvmvn"
Oct 31 00:30:44.151386 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:30:44.151346    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:31:14.151637 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:31:14.151600    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:31:44.151627 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:31:44.151587    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:32:14.151926 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:32:14.151883    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:32:44.150847 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:32:44.150812    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:33:14.151177 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:33:14.151140    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:33:44.151263 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:33:44.151232    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:33:47.950643 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:33:47.950614    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:33:47.952644 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:33:47.952619    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:34:14.152201 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:34:14.152158    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:34:44.152078 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:34:44.152024    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:35:09.386283 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.386249    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"]
Oct 31 00:35:09.386763 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.386574    4935 cpu_manager.go:414] "containerMap: removing container" podUID="131dbd11-5c4f-4d96-81e9-86a13ab4ae0c" containerName="insights-gathering"
Oct 31 00:35:09.386763 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.386585    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131dbd11-5c4f-4d96-81e9-86a13ab4ae0c" containerName="insights-gathering"
Oct 31 00:35:09.386763 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.386670    4935 cpu_manager.go:414] "containerMap: removing container" podUID="7e139718-2a1a-4b8f-b228-c8ceeda7fd70" containerName="insights-gathering"
Oct 31 00:35:09.386763 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.386676    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e139718-2a1a-4b8f-b228-c8ceeda7fd70" containerName="insights-gathering"
Oct 31 00:35:09.386763 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.386734    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e139718-2a1a-4b8f-b228-c8ceeda7fd70" containerName="insights-gathering"
Oct 31 00:35:09.386763 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.386743    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="131dbd11-5c4f-4d96-81e9-86a13ab4ae0c" containerName="insights-gathering"
Oct 31 00:35:09.388381 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.388365    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.466270 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.466237    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"]
Oct 31 00:35:09.469184 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.469156    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bf8cc1d-391c-4f42-8127-7f9d243062e8-cert\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.469339 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.469195    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3bf8cc1d-391c-4f42-8127-7f9d243062e8-manager-config\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.469339 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.469220    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bf8cc1d-391c-4f42-8127-7f9d243062e8-metrics-cert\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.469339 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.469295    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxn7\" (UniqueName: \"kubernetes.io/projected/3bf8cc1d-391c-4f42-8127-7f9d243062e8-kube-api-access-ptxn7\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.550160 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.550124    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 31 00:35:09.550336 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.550217    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:35:09.550336 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.550246    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 31 00:35:09.550336 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.550261    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 31 00:35:09.550336 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.550261    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 31 00:35:09.550336 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.550276    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7fllw\""
Oct 31 00:35:09.570271 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.570248    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bf8cc1d-391c-4f42-8127-7f9d243062e8-cert\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.570337 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.570283    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3bf8cc1d-391c-4f42-8127-7f9d243062e8-manager-config\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.570337 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.570308    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bf8cc1d-391c-4f42-8127-7f9d243062e8-metrics-cert\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.570398 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.570340    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxn7\" (UniqueName: \"kubernetes.io/projected/3bf8cc1d-391c-4f42-8127-7f9d243062e8-kube-api-access-ptxn7\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.570988 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.570959    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3bf8cc1d-391c-4f42-8127-7f9d243062e8-manager-config\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.572996 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.572974    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bf8cc1d-391c-4f42-8127-7f9d243062e8-cert\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.573096 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.572974    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bf8cc1d-391c-4f42-8127-7f9d243062e8-metrics-cert\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.630418 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.630384    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxn7\" (UniqueName: \"kubernetes.io/projected/3bf8cc1d-391c-4f42-8127-7f9d243062e8-kube-api-access-ptxn7\") pod \"lws-controller-manager-c99f78588-hcts7\" (UID: \"3bf8cc1d-391c-4f42-8127-7f9d243062e8\") " pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:09.698392 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:09.698306    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:10.074378 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:10.074346    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"]
Oct 31 00:35:10.080791 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:35:10.080755    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf8cc1d_391c_4f42_8127_7f9d243062e8.slice/crio-0a31beeb061727094f402c1c56dd32b626bdf64a4ad23309c9b5d10af6c1fd7e WatchSource:0}: Error finding container 0a31beeb061727094f402c1c56dd32b626bdf64a4ad23309c9b5d10af6c1fd7e: Status 404 returned error can't find the container with id 0a31beeb061727094f402c1c56dd32b626bdf64a4ad23309c9b5d10af6c1fd7e
Oct 31 00:35:10.082599 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:10.082583    4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:35:10.626072 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:10.626030    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7" event={"ID":"3bf8cc1d-391c-4f42-8127-7f9d243062e8","Type":"ContainerStarted","Data":"0a31beeb061727094f402c1c56dd32b626bdf64a4ad23309c9b5d10af6c1fd7e"}
Oct 31 00:35:14.156216 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:35:14.156180    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:35:14.645443 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:14.645391    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7" event={"ID":"3bf8cc1d-391c-4f42-8127-7f9d243062e8","Type":"ContainerStarted","Data":"3e4c2b6f636a56a9eed10f38b48f34764e54f8e30f23221178ea7b16bb891ed7"}
Oct 31 00:35:14.645624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:14.645471    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:14.764064 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:14.761277    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7" podStartSLOduration=1.541061368 podStartE2EDuration="5.76125817s" totalImagesPullingTime="4.220196802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:14.760821154 +0000 UTC m=+386.943350447" watchObservedRunningTime="2026-10-31 00:35:14.76125817 +0000 UTC m=+386.943787464"
Oct 31 00:35:25.651979 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:25.651945    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-c99f78588-hcts7"
Oct 31 00:35:44.151524 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:35:44.151485    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:35:50.630429 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:50.630391    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7"]
Oct 31 00:35:50.635353 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:50.635319    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7"
Oct 31 00:35:50.692844 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:50.692808    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xjj5q\""
Oct 31 00:35:50.699733 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:50.699708    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 00:35:50.699851 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:50.699800    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 00:35:50.699905 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:50.699890    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 00:35:50.728449 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:50.728414    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fdf\" (UniqueName: \"kubernetes.io/projected/a17d7101-2b60-4959-bb10-dcc468e05eb4-kube-api-access-h7fdf\") pod \"dns-operator-controller-manager-68df894cb4-hn7t7\" (UID: \"a17d7101-2b60-4959-bb10-dcc468e05eb4\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7"
Oct 31 00:35:50.735644 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:50.735616    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7"]
Oct 31 00:35:50.829561 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:50.829527    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fdf\" (UniqueName: \"kubernetes.io/projected/a17d7101-2b60-4959-bb10-dcc468e05eb4-kube-api-access-h7fdf\") pod \"dns-operator-controller-manager-68df894cb4-hn7t7\" (UID: \"a17d7101-2b60-4959-bb10-dcc468e05eb4\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7"
Oct 31 00:35:50.897027 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:50.896956    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fdf\" (UniqueName: \"kubernetes.io/projected/a17d7101-2b60-4959-bb10-dcc468e05eb4-kube-api-access-h7fdf\") pod \"dns-operator-controller-manager-68df894cb4-hn7t7\" (UID: \"a17d7101-2b60-4959-bb10-dcc468e05eb4\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7"
Oct 31 00:35:50.947185 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:50.947149    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7"
Oct 31 00:35:51.331579 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:51.331542    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7"]
Oct 31 00:35:51.338830 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:35:51.338806    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17d7101_2b60_4959_bb10_dcc468e05eb4.slice/crio-835fe978e129c183bf1dc5607bce78086538bc66d8e58366295d8f4fa3802ba5 WatchSource:0}: Error finding container 835fe978e129c183bf1dc5607bce78086538bc66d8e58366295d8f4fa3802ba5: Status 404 returned error can't find the container with id 835fe978e129c183bf1dc5607bce78086538bc66d8e58366295d8f4fa3802ba5
Oct 31 00:35:51.798770 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:51.798729    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7" event={"ID":"a17d7101-2b60-4959-bb10-dcc468e05eb4","Type":"ContainerStarted","Data":"835fe978e129c183bf1dc5607bce78086538bc66d8e58366295d8f4fa3802ba5"}
Oct 31 00:35:55.819936 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:55.819901    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7" event={"ID":"a17d7101-2b60-4959-bb10-dcc468e05eb4","Type":"ContainerStarted","Data":"b00e4d1a67e4e7a5cd754029fd20e481b73b60fa5b1b8b810d39369358a6b5a8"}
Oct 31 00:35:55.820391 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:55.819976    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7"
Oct 31 00:35:55.936773 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:35:55.936725    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7" podStartSLOduration=1.932518116 podStartE2EDuration="5.936711521s" totalImagesPullingTime="4.004193405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:55.934898373 +0000 UTC m=+428.117427669" watchObservedRunningTime="2026-10-31 00:35:55.936711521 +0000 UTC m=+428.119240817"
Oct 31 00:36:01.123237 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.123204    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"]
Oct 31 00:36:01.126422 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.126403    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"
Oct 31 00:36:01.177881 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.177844    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5m87r\""
Oct 31 00:36:01.221421 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.221383    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psl7n\" (UniqueName: \"kubernetes.io/projected/f2f4fcd6-6bde-4ad9-84e0-6519fc08848d-kube-api-access-psl7n\") pod \"kuadrant-console-plugin-66c4fb575d-n9j29\" (UID: \"f2f4fcd6-6bde-4ad9-84e0-6519fc08848d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"
Oct 31 00:36:01.221594 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.221460    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2f4fcd6-6bde-4ad9-84e0-6519fc08848d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n9j29\" (UID: \"f2f4fcd6-6bde-4ad9-84e0-6519fc08848d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"
Oct 31 00:36:01.221594 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.221504    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f2f4fcd6-6bde-4ad9-84e0-6519fc08848d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n9j29\" (UID: \"f2f4fcd6-6bde-4ad9-84e0-6519fc08848d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"
Oct 31 00:36:01.224805 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.224783    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 00:36:01.224893 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.224880    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 00:36:01.226226 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.226204    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"]
Oct 31 00:36:01.322897 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.322858    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psl7n\" (UniqueName: \"kubernetes.io/projected/f2f4fcd6-6bde-4ad9-84e0-6519fc08848d-kube-api-access-psl7n\") pod \"kuadrant-console-plugin-66c4fb575d-n9j29\" (UID: \"f2f4fcd6-6bde-4ad9-84e0-6519fc08848d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"
Oct 31 00:36:01.323130 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.322920    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2f4fcd6-6bde-4ad9-84e0-6519fc08848d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n9j29\" (UID: \"f2f4fcd6-6bde-4ad9-84e0-6519fc08848d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"
Oct 31 00:36:01.323130 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.322945    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f2f4fcd6-6bde-4ad9-84e0-6519fc08848d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n9j29\" (UID: \"f2f4fcd6-6bde-4ad9-84e0-6519fc08848d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"
Oct 31 00:36:01.323554 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.323536    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f2f4fcd6-6bde-4ad9-84e0-6519fc08848d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-n9j29\" (UID: \"f2f4fcd6-6bde-4ad9-84e0-6519fc08848d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"
Oct 31 00:36:01.325662 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.325641    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2f4fcd6-6bde-4ad9-84e0-6519fc08848d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-n9j29\" (UID: \"f2f4fcd6-6bde-4ad9-84e0-6519fc08848d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"
Oct 31 00:36:01.385751 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.385672    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psl7n\" (UniqueName: \"kubernetes.io/projected/f2f4fcd6-6bde-4ad9-84e0-6519fc08848d-kube-api-access-psl7n\") pod \"kuadrant-console-plugin-66c4fb575d-n9j29\" (UID: \"f2f4fcd6-6bde-4ad9-84e0-6519fc08848d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"
Oct 31 00:36:01.453523 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.453491    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"
Oct 31 00:36:01.827773 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.827736    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29"]
Oct 31 00:36:01.834167 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:36:01.834132    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2f4fcd6_6bde_4ad9_84e0_6519fc08848d.slice/crio-fb3ad4f688b6a8967c530ac4c2bf9f312585c6e518c20198d89d42349619e895 WatchSource:0}: Error finding container fb3ad4f688b6a8967c530ac4c2bf9f312585c6e518c20198d89d42349619e895: Status 404 returned error can't find the container with id fb3ad4f688b6a8967c530ac4c2bf9f312585c6e518c20198d89d42349619e895
Oct 31 00:36:01.845848 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:01.845821    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29" event={"ID":"f2f4fcd6-6bde-4ad9-84e0-6519fc08848d","Type":"ContainerStarted","Data":"fb3ad4f688b6a8967c530ac4c2bf9f312585c6e518c20198d89d42349619e895"}
Oct 31 00:36:06.826450 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:06.826416    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-hn7t7"
Oct 31 00:36:07.873546 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:07.873459    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29" event={"ID":"f2f4fcd6-6bde-4ad9-84e0-6519fc08848d","Type":"ContainerStarted","Data":"4ee7afcafef24ee9f7b64437d1dcd7330d8f72e6c3632c10180e40a64bd7aa61"}
Oct 31 00:36:14.152856 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:36:14.152811    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:36:41.724160 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.724107    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-n9j29" podStartSLOduration=34.940550352 podStartE2EDuration="40.724092256s" totalImagesPullingTime="5.783541904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:07.990269988 +0000 UTC m=+440.172799282" watchObservedRunningTime="2026-10-31 00:36:41.724092256 +0000 UTC m=+473.906621551"
Oct 31 00:36:41.724698 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.724682    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"]
Oct 31 00:36:41.730156 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.730136    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:36:41.751775 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.751749    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xbnq\" (UniqueName: \"kubernetes.io/projected/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-kube-api-access-5xbnq\") pod \"limitador-limitador-74dfcfb894-kwmcn\" (UID: \"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:36:41.751951 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.751866    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-config-file\") pod \"limitador-limitador-74dfcfb894-kwmcn\" (UID: \"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:36:41.783360 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.783333    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 00:36:41.784229 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.784204    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"]
Oct 31 00:36:41.852240 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.852211    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-config-file\") pod \"limitador-limitador-74dfcfb894-kwmcn\" (UID: \"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:36:41.852411 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.852254    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xbnq\" (UniqueName: \"kubernetes.io/projected/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-kube-api-access-5xbnq\") pod \"limitador-limitador-74dfcfb894-kwmcn\" (UID: \"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:36:41.852808 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.852788    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-config-file\") pod \"limitador-limitador-74dfcfb894-kwmcn\" (UID: \"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:36:41.913049 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.913011    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xbnq\" (UniqueName: \"kubernetes.io/projected/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-kube-api-access-5xbnq\") pod \"limitador-limitador-74dfcfb894-kwmcn\" (UID: \"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:36:41.954750 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:41.954725    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"]
Oct 31 00:36:42.041731 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:42.041697    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:36:42.417322 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:42.417242    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"]
Oct 31 00:36:42.423685 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:36:42.423655    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2efeeb_1ad7_412c_8a58_b7e5a84b8591.slice/crio-684eb3c0ae2269f1b2d281657ab1e69151933d847e1fea1d42655aee6a022cda WatchSource:0}: Error finding container 684eb3c0ae2269f1b2d281657ab1e69151933d847e1fea1d42655aee6a022cda: Status 404 returned error can't find the container with id 684eb3c0ae2269f1b2d281657ab1e69151933d847e1fea1d42655aee6a022cda
Oct 31 00:36:43.021675 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:43.021641    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn" event={"ID":"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591","Type":"ContainerStarted","Data":"684eb3c0ae2269f1b2d281657ab1e69151933d847e1fea1d42655aee6a022cda"}
Oct 31 00:36:44.153557 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:36:44.153519    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:36:45.031284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:45.031245    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn" event={"ID":"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591","Type":"ContainerStarted","Data":"7fdc1558115d106e5c6157c5aeda1891f6ea2b537455d1013fb42bbe162ed9cf"}
Oct 31 00:36:45.031492 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:45.031347    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:36:45.143954 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:45.143903    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn" podStartSLOduration=2.048622252 podStartE2EDuration="4.143890521s" totalImagesPullingTime="2.095268269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:45.143495139 +0000 UTC m=+477.326024432" watchObservedRunningTime="2026-10-31 00:36:45.143890521 +0000 UTC m=+477.326419833"
Oct 31 00:36:56.037453 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:36:56.037423    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:37:00.557112 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:00.557021    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"]
Oct 31 00:37:00.557500 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:00.557231    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn" podUID="ca2efeeb-1ad7-412c-8a58-b7e5a84b8591" containerName="limitador" containerID="cri-o://7fdc1558115d106e5c6157c5aeda1891f6ea2b537455d1013fb42bbe162ed9cf" gracePeriod=30
Oct 31 00:37:01.102676 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:01.102643    4935 generic.go:356] "Generic (PLEG): container finished" podID="ca2efeeb-1ad7-412c-8a58-b7e5a84b8591" containerID="7fdc1558115d106e5c6157c5aeda1891f6ea2b537455d1013fb42bbe162ed9cf" exitCode=0
Oct 31 00:37:01.102877 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:01.102726    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn" event={"ID":"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591","Type":"ContainerDied","Data":"7fdc1558115d106e5c6157c5aeda1891f6ea2b537455d1013fb42bbe162ed9cf"}
Oct 31 00:37:01.102877 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:01.102794    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn" event={"ID":"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591","Type":"ContainerDied","Data":"684eb3c0ae2269f1b2d281657ab1e69151933d847e1fea1d42655aee6a022cda"}
Oct 31 00:37:01.102877 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:01.102818    4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="684eb3c0ae2269f1b2d281657ab1e69151933d847e1fea1d42655aee6a022cda"
Oct 31 00:37:01.109081 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:01.109058    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:37:01.214129 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:01.213996    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-config-file\" (UniqueName: \"kubernetes.io/configmap/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-config-file\") pod \"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591\" (UID: \"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591\") "
Oct 31 00:37:01.214129 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:01.214099    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-kube-api-access-5xbnq\" (UniqueName: \"kubernetes.io/projected/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-kube-api-access-5xbnq\") pod \"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591\" (UID: \"ca2efeeb-1ad7-412c-8a58-b7e5a84b8591\") "
Oct 31 00:37:01.214464 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:01.214439    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-config-file" pod "ca2efeeb-1ad7-412c-8a58-b7e5a84b8591" (UID: "ca2efeeb-1ad7-412c-8a58-b7e5a84b8591"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:37:01.216506 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:01.216486    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-kube-api-access-5xbnq" pod "ca2efeeb-1ad7-412c-8a58-b7e5a84b8591" (UID: "ca2efeeb-1ad7-412c-8a58-b7e5a84b8591"). InnerVolumeSpecName "kube-api-access-5xbnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:37:01.315143 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:01.315110    4935 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-config-file\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:01.315143 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:01.315137    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xbnq\" (UniqueName: \"kubernetes.io/projected/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591-kube-api-access-5xbnq\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:02.107288 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:02.107259    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"
Oct 31 00:37:02.268483 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:02.268444    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"]
Oct 31 00:37:02.315469 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:02.315432    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kwmcn"]
Oct 31 00:37:03.942767 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:03.942732    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2efeeb-1ad7-412c-8a58-b7e5a84b8591" path="/var/lib/kubelet/pods/ca2efeeb-1ad7-412c-8a58-b7e5a84b8591/volumes"
Oct 31 00:37:14.151842 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:37:14.151805    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:37:44.151641 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:37:44.151595    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:37:54.120069 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.120015    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"]
Oct 31 00:37:54.120537 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.120522    4935 cpu_manager.go:414] "containerMap: removing container" podUID="ca2efeeb-1ad7-412c-8a58-b7e5a84b8591" containerName="limitador"
Oct 31 00:37:54.120583 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.120539    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2efeeb-1ad7-412c-8a58-b7e5a84b8591" containerName="limitador"
Oct 31 00:37:54.120663 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.120652    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2efeeb-1ad7-412c-8a58-b7e5a84b8591" containerName="limitador"
Oct 31 00:37:54.123938 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.123909    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:37:54.151119 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.151088    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplpf\" (UniqueName: \"kubernetes.io/projected/467ce148-368f-48b7-af64-860d27e8c308-kube-api-access-kplpf\") pod \"kserve-controller-manager-5bbfdbf8df-sqrvq\" (UID: \"467ce148-368f-48b7-af64-860d27e8c308\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:37:54.151254 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.151212    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/467ce148-368f-48b7-af64-860d27e8c308-cert\") pod \"kserve-controller-manager-5bbfdbf8df-sqrvq\" (UID: \"467ce148-368f-48b7-af64-860d27e8c308\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:37:54.227108 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.227074    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Oct 31 00:37:54.227108 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.227108    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 31 00:37:54.227319 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.227140    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 31 00:37:54.227319 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.227154    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-p8v2k\""
Oct 31 00:37:54.228910 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.228883    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"]
Oct 31 00:37:54.252305 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.252281    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/467ce148-368f-48b7-af64-860d27e8c308-cert\") pod \"kserve-controller-manager-5bbfdbf8df-sqrvq\" (UID: \"467ce148-368f-48b7-af64-860d27e8c308\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:37:54.252456 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.252334    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplpf\" (UniqueName: \"kubernetes.io/projected/467ce148-368f-48b7-af64-860d27e8c308-kube-api-access-kplpf\") pod \"kserve-controller-manager-5bbfdbf8df-sqrvq\" (UID: \"467ce148-368f-48b7-af64-860d27e8c308\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:37:54.254862 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.254839    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/467ce148-368f-48b7-af64-860d27e8c308-cert\") pod \"kserve-controller-manager-5bbfdbf8df-sqrvq\" (UID: \"467ce148-368f-48b7-af64-860d27e8c308\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:37:54.276301 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.276261    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-pj22z"]
Oct 31 00:37:54.279528 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.279513    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-pj22z"
Oct 31 00:37:54.312357 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.312331    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplpf\" (UniqueName: \"kubernetes.io/projected/467ce148-368f-48b7-af64-860d27e8c308-kube-api-access-kplpf\") pod \"kserve-controller-manager-5bbfdbf8df-sqrvq\" (UID: \"467ce148-368f-48b7-af64-860d27e8c308\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:37:54.323529 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.323505    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-pj22z"]
Oct 31 00:37:54.353001 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.352974    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/607a5b6f-e93a-4435-8b43-6ac00afa77a6-data\") pod \"seaweedfs-78bb9c6ff8-pj22z\" (UID: \"607a5b6f-e93a-4435-8b43-6ac00afa77a6\") " pod="kserve/seaweedfs-78bb9c6ff8-pj22z"
Oct 31 00:37:54.353177 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.353005    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn9jx\" (UniqueName: \"kubernetes.io/projected/607a5b6f-e93a-4435-8b43-6ac00afa77a6-kube-api-access-wn9jx\") pod \"seaweedfs-78bb9c6ff8-pj22z\" (UID: \"607a5b6f-e93a-4435-8b43-6ac00afa77a6\") " pod="kserve/seaweedfs-78bb9c6ff8-pj22z"
Oct 31 00:37:54.382007 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.381938    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cwgh2\""
Oct 31 00:37:54.382007 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.381970    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 31 00:37:54.436785 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.436758    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:37:54.454025 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.454000    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/607a5b6f-e93a-4435-8b43-6ac00afa77a6-data\") pod \"seaweedfs-78bb9c6ff8-pj22z\" (UID: \"607a5b6f-e93a-4435-8b43-6ac00afa77a6\") " pod="kserve/seaweedfs-78bb9c6ff8-pj22z"
Oct 31 00:37:54.454158 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.454047    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn9jx\" (UniqueName: \"kubernetes.io/projected/607a5b6f-e93a-4435-8b43-6ac00afa77a6-kube-api-access-wn9jx\") pod \"seaweedfs-78bb9c6ff8-pj22z\" (UID: \"607a5b6f-e93a-4435-8b43-6ac00afa77a6\") " pod="kserve/seaweedfs-78bb9c6ff8-pj22z"
Oct 31 00:37:54.454397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.454378    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/607a5b6f-e93a-4435-8b43-6ac00afa77a6-data\") pod \"seaweedfs-78bb9c6ff8-pj22z\" (UID: \"607a5b6f-e93a-4435-8b43-6ac00afa77a6\") " pod="kserve/seaweedfs-78bb9c6ff8-pj22z"
Oct 31 00:37:54.515735 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.514606    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn9jx\" (UniqueName: \"kubernetes.io/projected/607a5b6f-e93a-4435-8b43-6ac00afa77a6-kube-api-access-wn9jx\") pod \"seaweedfs-78bb9c6ff8-pj22z\" (UID: \"607a5b6f-e93a-4435-8b43-6ac00afa77a6\") " pod="kserve/seaweedfs-78bb9c6ff8-pj22z"
Oct 31 00:37:54.591734 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.591682    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-pj22z"
Oct 31 00:37:54.808091 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.808058    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"]
Oct 31 00:37:54.814773 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:37:54.814746    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467ce148_368f_48b7_af64_860d27e8c308.slice/crio-a67412b464bd31f1c5b811f7e3a52c9f2ceef3e369dd69790636b75ecc3d10ea WatchSource:0}: Error finding container a67412b464bd31f1c5b811f7e3a52c9f2ceef3e369dd69790636b75ecc3d10ea: Status 404 returned error can't find the container with id a67412b464bd31f1c5b811f7e3a52c9f2ceef3e369dd69790636b75ecc3d10ea
Oct 31 00:37:54.960622 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:54.960537    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-pj22z"]
Oct 31 00:37:54.966974 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:37:54.966948    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607a5b6f_e93a_4435_8b43_6ac00afa77a6.slice/crio-57e521bcd3d7840b0d80e4655549c5e362b26e0d4e0de0a7fa8ac35b0e02dabc WatchSource:0}: Error finding container 57e521bcd3d7840b0d80e4655549c5e362b26e0d4e0de0a7fa8ac35b0e02dabc: Status 404 returned error can't find the container with id 57e521bcd3d7840b0d80e4655549c5e362b26e0d4e0de0a7fa8ac35b0e02dabc
Oct 31 00:37:55.328680 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:55.328647    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-pj22z" event={"ID":"607a5b6f-e93a-4435-8b43-6ac00afa77a6","Type":"ContainerStarted","Data":"57e521bcd3d7840b0d80e4655549c5e362b26e0d4e0de0a7fa8ac35b0e02dabc"}
Oct 31 00:37:55.329780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:37:55.329755    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq" event={"ID":"467ce148-368f-48b7-af64-860d27e8c308","Type":"ContainerStarted","Data":"a67412b464bd31f1c5b811f7e3a52c9f2ceef3e369dd69790636b75ecc3d10ea"}
Oct 31 00:38:01.361403 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:01.361361    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-pj22z" event={"ID":"607a5b6f-e93a-4435-8b43-6ac00afa77a6","Type":"ContainerStarted","Data":"e3722b43f7a8ea8f33bde3e23027e53d515ac5f6e9d0a65a8f4250ea5d150577"}
Oct 31 00:38:01.361775 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:01.361428    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-78bb9c6ff8-pj22z"
Oct 31 00:38:01.473922 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:01.473864    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-78bb9c6ff8-pj22z" podStartSLOduration=1.6517324850000001 podStartE2EDuration="7.473850056s" totalImagesPullingTime="5.822117571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:01.473419632 +0000 UTC m=+553.655948930" watchObservedRunningTime="2026-10-31 00:38:01.473850056 +0000 UTC m=+553.656379349"
Oct 31 00:38:03.371357 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:03.371320    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq" event={"ID":"467ce148-368f-48b7-af64-860d27e8c308","Type":"ContainerStarted","Data":"45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d"}
Oct 31 00:38:03.371844 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:03.371461    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:38:03.490437 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:03.490380    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq" podStartSLOduration=1.9022756360000002 podStartE2EDuration="9.490368223s" totalImagesPullingTime="7.588092587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:03.488098561 +0000 UTC m=+555.670627854" watchObservedRunningTime="2026-10-31 00:38:03.490368223 +0000 UTC m=+555.672897552"
Oct 31 00:38:07.368409 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:07.368380    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-78bb9c6ff8-pj22z"
Oct 31 00:38:14.153110 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:38:14.152992    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:38:34.381545 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:34.381464    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:38:44.152529 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:38:44.152493    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:38:47.984202 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:47.984173    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:38:47.984773 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:47.984499    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:38:50.596318 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.596287    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"]
Oct 31 00:38:50.596788 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.596484    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq" podUID="467ce148-368f-48b7-af64-860d27e8c308" containerName="manager" containerID="cri-o://45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d" gracePeriod=10
Oct 31 00:38:50.627271 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.627244    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5bbfdbf8df-zrb48"]
Oct 31 00:38:50.630505 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.630489    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48"
Oct 31 00:38:50.703436 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.703410    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5bbfdbf8df-zrb48"]
Oct 31 00:38:50.722994 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.722965    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1671631e-392e-4378-81ee-f620b3388b3c-cert\") pod \"kserve-controller-manager-5bbfdbf8df-zrb48\" (UID: \"1671631e-392e-4378-81ee-f620b3388b3c\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48"
Oct 31 00:38:50.723168 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.723009    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gcrd\" (UniqueName: \"kubernetes.io/projected/1671631e-392e-4378-81ee-f620b3388b3c-kube-api-access-8gcrd\") pod \"kserve-controller-manager-5bbfdbf8df-zrb48\" (UID: \"1671631e-392e-4378-81ee-f620b3388b3c\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48"
Oct 31 00:38:50.823809 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.823771    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gcrd\" (UniqueName: \"kubernetes.io/projected/1671631e-392e-4378-81ee-f620b3388b3c-kube-api-access-8gcrd\") pod \"kserve-controller-manager-5bbfdbf8df-zrb48\" (UID: \"1671631e-392e-4378-81ee-f620b3388b3c\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48"
Oct 31 00:38:50.823934 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.823914    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1671631e-392e-4378-81ee-f620b3388b3c-cert\") pod \"kserve-controller-manager-5bbfdbf8df-zrb48\" (UID: \"1671631e-392e-4378-81ee-f620b3388b3c\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48"
Oct 31 00:38:50.826494 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.826470    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1671631e-392e-4378-81ee-f620b3388b3c-cert\") pod \"kserve-controller-manager-5bbfdbf8df-zrb48\" (UID: \"1671631e-392e-4378-81ee-f620b3388b3c\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48"
Oct 31 00:38:50.843100 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.843078    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:38:50.884843 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.884777    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gcrd\" (UniqueName: \"kubernetes.io/projected/1671631e-392e-4378-81ee-f620b3388b3c-kube-api-access-8gcrd\") pod \"kserve-controller-manager-5bbfdbf8df-zrb48\" (UID: \"1671631e-392e-4378-81ee-f620b3388b3c\") " pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48"
Oct 31 00:38:50.924271 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.924242    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/467ce148-368f-48b7-af64-860d27e8c308-cert\" (UniqueName: \"kubernetes.io/secret/467ce148-368f-48b7-af64-860d27e8c308-cert\") pod \"467ce148-368f-48b7-af64-860d27e8c308\" (UID: \"467ce148-368f-48b7-af64-860d27e8c308\") "
Oct 31 00:38:50.924397 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.924292    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467ce148-368f-48b7-af64-860d27e8c308-kube-api-access-kplpf\" (UniqueName: \"kubernetes.io/projected/467ce148-368f-48b7-af64-860d27e8c308-kube-api-access-kplpf\") pod \"467ce148-368f-48b7-af64-860d27e8c308\" (UID: \"467ce148-368f-48b7-af64-860d27e8c308\") "
Oct 31 00:38:50.926538 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.926511    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467ce148-368f-48b7-af64-860d27e8c308-kube-api-access-kplpf" pod "467ce148-368f-48b7-af64-860d27e8c308" (UID: "467ce148-368f-48b7-af64-860d27e8c308"). InnerVolumeSpecName "kube-api-access-kplpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:38:50.926538 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.926518    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/467ce148-368f-48b7-af64-860d27e8c308-cert" pod "467ce148-368f-48b7-af64-860d27e8c308" (UID: "467ce148-368f-48b7-af64-860d27e8c308"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:38:50.982211 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:50.982165    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48"
Oct 31 00:38:51.025810 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.025779    4935 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/467ce148-368f-48b7-af64-860d27e8c308-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:38:51.025996 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.025985    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kplpf\" (UniqueName: \"kubernetes.io/projected/467ce148-368f-48b7-af64-860d27e8c308-kube-api-access-kplpf\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:38:51.356392 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.356357    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5bbfdbf8df-zrb48"]
Oct 31 00:38:51.362702 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:38:51.362669    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1671631e_392e_4378_81ee_f620b3388b3c.slice/crio-6f777234ea5feffcb341a12b4c424b69b3bc18c48309dd05e1fe2aaf4abd165e WatchSource:0}: Error finding container 6f777234ea5feffcb341a12b4c424b69b3bc18c48309dd05e1fe2aaf4abd165e: Status 404 returned error can't find the container with id 6f777234ea5feffcb341a12b4c424b69b3bc18c48309dd05e1fe2aaf4abd165e
Oct 31 00:38:51.574688 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.574653    4935 generic.go:356] "Generic (PLEG): container finished" podID="467ce148-368f-48b7-af64-860d27e8c308" containerID="45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d" exitCode=0
Oct 31 00:38:51.574885 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.574708    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"
Oct 31 00:38:51.574885 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.574749    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq" event={"ID":"467ce148-368f-48b7-af64-860d27e8c308","Type":"ContainerDied","Data":"45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d"}
Oct 31 00:38:51.574885 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.574810    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bbfdbf8df-sqrvq" event={"ID":"467ce148-368f-48b7-af64-860d27e8c308","Type":"ContainerDied","Data":"a67412b464bd31f1c5b811f7e3a52c9f2ceef3e369dd69790636b75ecc3d10ea"}
Oct 31 00:38:51.574885 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.574841    4935 scope.go:122] "RemoveContainer" containerID="45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d"
Oct 31 00:38:51.575958 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.575938    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48" event={"ID":"1671631e-392e-4378-81ee-f620b3388b3c","Type":"ContainerStarted","Data":"6f777234ea5feffcb341a12b4c424b69b3bc18c48309dd05e1fe2aaf4abd165e"}
Oct 31 00:38:51.583456 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.583442    4935 scope.go:122] "RemoveContainer" containerID="45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d"
Oct 31 00:38:51.583714 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:38:51.583694    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d\": container with ID starting with 45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d not found: ID does not exist" containerID="45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d"
Oct 31 00:38:51.583750 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.583726    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d"} err="failed to get container status \"45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d\": rpc error: code = NotFound desc = could not find container \"45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d\": container with ID starting with 45b2cf989338607ea86ae227bda708e7374091867955aca2f6436d026b76d39d not found: ID does not exist"
Oct 31 00:38:51.734084 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.734050    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"]
Oct 31 00:38:51.740347 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.740322    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5bbfdbf8df-sqrvq"]
Oct 31 00:38:51.942954 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:51.942864    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467ce148-368f-48b7-af64-860d27e8c308" path="/var/lib/kubelet/pods/467ce148-368f-48b7-af64-860d27e8c308/volumes"
Oct 31 00:38:52.582711 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:52.582678    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48" event={"ID":"1671631e-392e-4378-81ee-f620b3388b3c","Type":"ContainerStarted","Data":"253dc484904cfcfdd2d8d5bf3c4cd5156e5200fef186f9dd07099e58cb75098f"}
Oct 31 00:38:52.582847 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:52.582822    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48"
Oct 31 00:38:52.695349 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:38:52.695296    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48" podStartSLOduration=1.550658965 podStartE2EDuration="2.695282934s" totalImagesPullingTime="1.144623969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:52.694767158 +0000 UTC m=+604.877296451" watchObservedRunningTime="2026-10-31 00:38:52.695282934 +0000 UTC m=+604.877812239"
Oct 31 00:39:14.152929 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:39:14.152884    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:39:23.592757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:23.592728    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5bbfdbf8df-zrb48"
Oct 31 00:39:25.982697 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:25.982662    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-56ddc4c448-kh7wx"]
Oct 31 00:39:25.983236 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:25.983218    4935 cpu_manager.go:414] "containerMap: removing container" podUID="467ce148-368f-48b7-af64-860d27e8c308" containerName="manager"
Oct 31 00:39:25.983314 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:25.983240    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467ce148-368f-48b7-af64-860d27e8c308" containerName="manager"
Oct 31 00:39:25.983369 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:25.983332    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="467ce148-368f-48b7-af64-860d27e8c308" containerName="manager"
Oct 31 00:39:25.986549 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:25.986528    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-kh7wx"
Oct 31 00:39:26.040578 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.040550    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-mk2ht"]
Oct 31 00:39:26.044421 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.044395    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-kh7wx"]
Oct 31 00:39:26.044546 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.044502    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:26.085703 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.085678    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-rgcq7\""
Oct 31 00:39:26.085703 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.085702    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Oct 31 00:39:26.099683 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.099658    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-mk2ht"]
Oct 31 00:39:26.119497 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.119469    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e3988a-5c8a-4aa6-8b25-a9ba12a81957-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-mk2ht\" (UID: \"20e3988a-5c8a-4aa6-8b25-a9ba12a81957\") " pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:26.119640 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.119515    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn242\" (UniqueName: \"kubernetes.io/projected/20e3988a-5c8a-4aa6-8b25-a9ba12a81957-kube-api-access-wn242\") pod \"odh-model-controller-7f9f8cdc65-mk2ht\" (UID: \"20e3988a-5c8a-4aa6-8b25-a9ba12a81957\") " pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:26.119682 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.119644    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9744b759-f7b0-4a9a-b3eb-b9784910cdc9-tls-certs\") pod \"model-serving-api-56ddc4c448-kh7wx\" (UID: \"9744b759-f7b0-4a9a-b3eb-b9784910cdc9\") " pod="kserve/model-serving-api-56ddc4c448-kh7wx"
Oct 31 00:39:26.119725 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.119697    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20e3988a-5c8a-4aa6-8b25-a9ba12a81957-cert\") pod \"odh-model-controller-7f9f8cdc65-mk2ht\" (UID: \"20e3988a-5c8a-4aa6-8b25-a9ba12a81957\") " pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:26.119812 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.119796    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kzht\" (UniqueName: \"kubernetes.io/projected/9744b759-f7b0-4a9a-b3eb-b9784910cdc9-kube-api-access-4kzht\") pod \"model-serving-api-56ddc4c448-kh7wx\" (UID: \"9744b759-f7b0-4a9a-b3eb-b9784910cdc9\") " pod="kserve/model-serving-api-56ddc4c448-kh7wx"
Oct 31 00:39:26.141633 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.141602    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Oct 31 00:39:26.141769 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.141674    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-prwzz\""
Oct 31 00:39:26.141769 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.141737    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Oct 31 00:39:26.220800 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.220770    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kzht\" (UniqueName: \"kubernetes.io/projected/9744b759-f7b0-4a9a-b3eb-b9784910cdc9-kube-api-access-4kzht\") pod \"model-serving-api-56ddc4c448-kh7wx\" (UID: \"9744b759-f7b0-4a9a-b3eb-b9784910cdc9\") " pod="kserve/model-serving-api-56ddc4c448-kh7wx"
Oct 31 00:39:26.220976 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.220824    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e3988a-5c8a-4aa6-8b25-a9ba12a81957-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-mk2ht\" (UID: \"20e3988a-5c8a-4aa6-8b25-a9ba12a81957\") " pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:26.220976 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.220848    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn242\" (UniqueName: \"kubernetes.io/projected/20e3988a-5c8a-4aa6-8b25-a9ba12a81957-kube-api-access-wn242\") pod \"odh-model-controller-7f9f8cdc65-mk2ht\" (UID: \"20e3988a-5c8a-4aa6-8b25-a9ba12a81957\") " pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:26.220976 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.220882    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9744b759-f7b0-4a9a-b3eb-b9784910cdc9-tls-certs\") pod \"model-serving-api-56ddc4c448-kh7wx\" (UID: \"9744b759-f7b0-4a9a-b3eb-b9784910cdc9\") " pod="kserve/model-serving-api-56ddc4c448-kh7wx"
Oct 31 00:39:26.220976 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.220905    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20e3988a-5c8a-4aa6-8b25-a9ba12a81957-cert\") pod \"odh-model-controller-7f9f8cdc65-mk2ht\" (UID: \"20e3988a-5c8a-4aa6-8b25-a9ba12a81957\") " pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:26.223757 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.223734    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e3988a-5c8a-4aa6-8b25-a9ba12a81957-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-mk2ht\" (UID: \"20e3988a-5c8a-4aa6-8b25-a9ba12a81957\") " pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:26.223873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.223799    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20e3988a-5c8a-4aa6-8b25-a9ba12a81957-cert\") pod \"odh-model-controller-7f9f8cdc65-mk2ht\" (UID: \"20e3988a-5c8a-4aa6-8b25-a9ba12a81957\") " pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:26.223873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.223799    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9744b759-f7b0-4a9a-b3eb-b9784910cdc9-tls-certs\") pod \"model-serving-api-56ddc4c448-kh7wx\" (UID: \"9744b759-f7b0-4a9a-b3eb-b9784910cdc9\") " pod="kserve/model-serving-api-56ddc4c448-kh7wx"
Oct 31 00:39:26.281714 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.281681    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn242\" (UniqueName: \"kubernetes.io/projected/20e3988a-5c8a-4aa6-8b25-a9ba12a81957-kube-api-access-wn242\") pod \"odh-model-controller-7f9f8cdc65-mk2ht\" (UID: \"20e3988a-5c8a-4aa6-8b25-a9ba12a81957\") " pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:26.281879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.281803    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kzht\" (UniqueName: \"kubernetes.io/projected/9744b759-f7b0-4a9a-b3eb-b9784910cdc9-kube-api-access-4kzht\") pod \"model-serving-api-56ddc4c448-kh7wx\" (UID: \"9744b759-f7b0-4a9a-b3eb-b9784910cdc9\") " pod="kserve/model-serving-api-56ddc4c448-kh7wx"
Oct 31 00:39:26.299768 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.299744    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-kh7wx"
Oct 31 00:39:26.357454 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.357418    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:26.671427 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.671346    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-kh7wx"]
Oct 31 00:39:26.677790 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:39:26.677761    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9744b759_f7b0_4a9a_b3eb_b9784910cdc9.slice/crio-43329283b8edc83d0ac9b8841d6e416f7786d9a368de1b49c1e0bfe34eb54d37 WatchSource:0}: Error finding container 43329283b8edc83d0ac9b8841d6e416f7786d9a368de1b49c1e0bfe34eb54d37: Status 404 returned error can't find the container with id 43329283b8edc83d0ac9b8841d6e416f7786d9a368de1b49c1e0bfe34eb54d37
Oct 31 00:39:26.725572 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.725539    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-kh7wx" event={"ID":"9744b759-f7b0-4a9a-b3eb-b9784910cdc9","Type":"ContainerStarted","Data":"43329283b8edc83d0ac9b8841d6e416f7786d9a368de1b49c1e0bfe34eb54d37"}
Oct 31 00:39:26.787394 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:26.787362    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-mk2ht"]
Oct 31 00:39:26.794021 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:39:26.793992    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e3988a_5c8a_4aa6_8b25_a9ba12a81957.slice/crio-506760828b50a0ef97c8347d10b9ae50f7ee34518593bb7b8e08a661ca60dd69 WatchSource:0}: Error finding container 506760828b50a0ef97c8347d10b9ae50f7ee34518593bb7b8e08a661ca60dd69: Status 404 returned error can't find the container with id 506760828b50a0ef97c8347d10b9ae50f7ee34518593bb7b8e08a661ca60dd69
Oct 31 00:39:27.730997 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:27.730961    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht" event={"ID":"20e3988a-5c8a-4aa6-8b25-a9ba12a81957","Type":"ContainerStarted","Data":"506760828b50a0ef97c8347d10b9ae50f7ee34518593bb7b8e08a661ca60dd69"}
Oct 31 00:39:29.744853 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:29.744811    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-kh7wx" event={"ID":"9744b759-f7b0-4a9a-b3eb-b9784910cdc9","Type":"ContainerStarted","Data":"e2c5275bc252361c63e60c30ebc5a97e058680c5769c446f18a8abc73951a5b9"}
Oct 31 00:39:29.745366 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:29.745124    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-56ddc4c448-kh7wx"
Oct 31 00:39:29.858636 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:29.858584    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/model-serving-api-56ddc4c448-kh7wx" podStartSLOduration=2.509368474 podStartE2EDuration="4.858571076s" totalImagesPullingTime="2.349202602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:29.858260064 +0000 UTC m=+642.040789358" watchObservedRunningTime="2026-10-31 00:39:29.858571076 +0000 UTC m=+642.041100369"
Oct 31 00:39:31.755292 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:31.755251    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht" event={"ID":"20e3988a-5c8a-4aa6-8b25-a9ba12a81957","Type":"ContainerStarted","Data":"bfec17ca7f95d1acd1ed97373b362e12f881df55d6e2cf7174042d9671d9b7d1"}
Oct 31 00:39:31.755722 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:31.755442    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:31.870931 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:31.870863    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht" podStartSLOduration=2.37016248 podStartE2EDuration="6.870850496s" totalImagesPullingTime="4.500688016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:31.869913283 +0000 UTC m=+644.052442578" watchObservedRunningTime="2026-10-31 00:39:31.870850496 +0000 UTC m=+644.053379853"
Oct 31 00:39:40.753331 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:40.753295    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-56ddc4c448-kh7wx"
Oct 31 00:39:42.761580 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:42.761549    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7f9f8cdc65-mk2ht"
Oct 31 00:39:44.152341 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:39:44.152302    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:39:44.248009 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.247971    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-init-26c2c"]
Oct 31 00:39:44.251578 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.251561    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-26c2c"
Oct 31 00:39:44.309334 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.309301    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-26c2c"]
Oct 31 00:39:44.354525 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.354493    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 31 00:39:44.373422 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.373390    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073e9ffb-99fb-4f43-8501-867b60b67286-model-cache\") pod \"s3-init-26c2c\" (UID: \"073e9ffb-99fb-4f43-8501-867b60b67286\") " pod="kserve/s3-init-26c2c"
Oct 31 00:39:44.373530 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.373439    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cgn6\" (UniqueName: \"kubernetes.io/projected/073e9ffb-99fb-4f43-8501-867b60b67286-kube-api-access-6cgn6\") pod \"s3-init-26c2c\" (UID: \"073e9ffb-99fb-4f43-8501-867b60b67286\") " pod="kserve/s3-init-26c2c"
Oct 31 00:39:44.474805 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.474723    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgn6\" (UniqueName: \"kubernetes.io/projected/073e9ffb-99fb-4f43-8501-867b60b67286-kube-api-access-6cgn6\") pod \"s3-init-26c2c\" (UID: \"073e9ffb-99fb-4f43-8501-867b60b67286\") " pod="kserve/s3-init-26c2c"
Oct 31 00:39:44.474805 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.474805    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073e9ffb-99fb-4f43-8501-867b60b67286-model-cache\") pod \"s3-init-26c2c\" (UID: \"073e9ffb-99fb-4f43-8501-867b60b67286\") " pod="kserve/s3-init-26c2c"
Oct 31 00:39:44.475172 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.475157    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073e9ffb-99fb-4f43-8501-867b60b67286-model-cache\") pod \"s3-init-26c2c\" (UID: \"073e9ffb-99fb-4f43-8501-867b60b67286\") " pod="kserve/s3-init-26c2c"
Oct 31 00:39:44.535783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.535744    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cgn6\" (UniqueName: \"kubernetes.io/projected/073e9ffb-99fb-4f43-8501-867b60b67286-kube-api-access-6cgn6\") pod \"s3-init-26c2c\" (UID: \"073e9ffb-99fb-4f43-8501-867b60b67286\") " pod="kserve/s3-init-26c2c"
Oct 31 00:39:44.565116 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.565091    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-26c2c"
Oct 31 00:39:44.935328 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:44.935296    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-26c2c"]
Oct 31 00:39:44.941729 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:39:44.941703    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod073e9ffb_99fb_4f43_8501_867b60b67286.slice/crio-6b272a4c099a83e812746155d966b305e2e79986213801f9276c771f7e9b9432 WatchSource:0}: Error finding container 6b272a4c099a83e812746155d966b305e2e79986213801f9276c771f7e9b9432: Status 404 returned error can't find the container with id 6b272a4c099a83e812746155d966b305e2e79986213801f9276c771f7e9b9432
Oct 31 00:39:45.816261 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:45.816227    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-26c2c" event={"ID":"073e9ffb-99fb-4f43-8501-867b60b67286","Type":"ContainerStarted","Data":"6b272a4c099a83e812746155d966b305e2e79986213801f9276c771f7e9b9432"}
Oct 31 00:39:54.251520 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:54.251495    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 31 00:39:54.862202 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:54.862167    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-26c2c" event={"ID":"073e9ffb-99fb-4f43-8501-867b60b67286","Type":"ContainerStarted","Data":"c17f92b20420c49585964c8dbd944724f8f782094fcb68dbe4e95cc4a79d5419"}
Oct 31 00:39:55.867646 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:55.867617    4935 generic.go:356] "Generic (PLEG): container finished" podID="073e9ffb-99fb-4f43-8501-867b60b67286" containerID="c17f92b20420c49585964c8dbd944724f8f782094fcb68dbe4e95cc4a79d5419" exitCode=0
Oct 31 00:39:55.868047 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:39:55.867653    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-26c2c" event={"ID":"073e9ffb-99fb-4f43-8501-867b60b67286","Type":"ContainerDied","Data":"c17f92b20420c49585964c8dbd944724f8f782094fcb68dbe4e95cc4a79d5419"}
Oct 31 00:40:00.896464 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:40:00.896423    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-26c2c" event={"ID":"073e9ffb-99fb-4f43-8501-867b60b67286","Type":"ContainerStarted","Data":"ed94b3bfb8e2da0f821ce36e5711d91dca32d24c6d3a14e85ca853c14da714b8"}
Oct 31 00:40:01.009403 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:40:01.009334    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-init-26c2c" podStartSLOduration=1.310129842 podStartE2EDuration="17.00931434s" totalImagesPullingTime="14.186001297s" totalInitContainerRuntime="1.513183201s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:01.008155881 +0000 UTC m=+673.190685174" watchObservedRunningTime="2026-10-31 00:40:01.00931434 +0000 UTC m=+673.191843680"
Oct 31 00:40:14.151835 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:40:14.151799    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:40:44.151381 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:40:44.151344    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:41:06.174737 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:06.174651    4935 generic.go:356] "Generic (PLEG): container finished" podID="073e9ffb-99fb-4f43-8501-867b60b67286" containerID="ed94b3bfb8e2da0f821ce36e5711d91dca32d24c6d3a14e85ca853c14da714b8" exitCode=0
Oct 31 00:41:06.174737 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:06.174728    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-26c2c" event={"ID":"073e9ffb-99fb-4f43-8501-867b60b67286","Type":"ContainerDied","Data":"ed94b3bfb8e2da0f821ce36e5711d91dca32d24c6d3a14e85ca853c14da714b8"}
Oct 31 00:41:07.309798 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:07.309772    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-26c2c"
Oct 31 00:41:07.371149 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:07.371117    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073e9ffb-99fb-4f43-8501-867b60b67286-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073e9ffb-99fb-4f43-8501-867b60b67286-model-cache\") pod \"073e9ffb-99fb-4f43-8501-867b60b67286\" (UID: \"073e9ffb-99fb-4f43-8501-867b60b67286\") "
Oct 31 00:41:07.371318 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:07.371216    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073e9ffb-99fb-4f43-8501-867b60b67286-kube-api-access-6cgn6\" (UniqueName: \"kubernetes.io/projected/073e9ffb-99fb-4f43-8501-867b60b67286-kube-api-access-6cgn6\") pod \"073e9ffb-99fb-4f43-8501-867b60b67286\" (UID: \"073e9ffb-99fb-4f43-8501-867b60b67286\") "
Oct 31 00:41:07.373477 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:07.373452    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073e9ffb-99fb-4f43-8501-867b60b67286-kube-api-access-6cgn6" pod "073e9ffb-99fb-4f43-8501-867b60b67286" (UID: "073e9ffb-99fb-4f43-8501-867b60b67286"). InnerVolumeSpecName "kube-api-access-6cgn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:41:07.393868 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:07.393835    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073e9ffb-99fb-4f43-8501-867b60b67286-model-cache" pod "073e9ffb-99fb-4f43-8501-867b60b67286" (UID: "073e9ffb-99fb-4f43-8501-867b60b67286"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:41:07.472823 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:07.472736    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073e9ffb-99fb-4f43-8501-867b60b67286-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:41:07.472823 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:07.472766    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cgn6\" (UniqueName: \"kubernetes.io/projected/073e9ffb-99fb-4f43-8501-867b60b67286-kube-api-access-6cgn6\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:41:08.185462 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:08.185437    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-26c2c"
Oct 31 00:41:08.185462 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:08.185441    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-26c2c" event={"ID":"073e9ffb-99fb-4f43-8501-867b60b67286","Type":"ContainerDied","Data":"6b272a4c099a83e812746155d966b305e2e79986213801f9276c771f7e9b9432"}
Oct 31 00:41:08.185678 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:08.185475    4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b272a4c099a83e812746155d966b305e2e79986213801f9276c771f7e9b9432"
Oct 31 00:41:14.152295 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:41:14.152252    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:41:39.192974 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.192886    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"]
Oct 31 00:41:39.193395 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.193306    4935 cpu_manager.go:414] "containerMap: removing container" podUID="073e9ffb-99fb-4f43-8501-867b60b67286" containerName="download-hf-model"
Oct 31 00:41:39.193395 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.193321    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073e9ffb-99fb-4f43-8501-867b60b67286" containerName="download-hf-model"
Oct 31 00:41:39.193395 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.193330    4935 cpu_manager.go:414] "containerMap: removing container" podUID="073e9ffb-99fb-4f43-8501-867b60b67286" containerName="s3-init"
Oct 31 00:41:39.193395 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.193335    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073e9ffb-99fb-4f43-8501-867b60b67286" containerName="s3-init"
Oct 31 00:41:39.193529 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.193432    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="073e9ffb-99fb-4f43-8501-867b60b67286" containerName="s3-init"
Oct 31 00:41:39.196591 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.196576    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.248898 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.248861    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"]
Oct 31 00:41:39.293955 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.293915    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"seaweedfs-s3-creds\""
Oct 31 00:41:39.294167 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.294060    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"default-dockercfg-5bpj4\""
Oct 31 00:41:39.294167 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.294061    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"cpu-accel-preset-kserve-self-signed-certs\""
Oct 31 00:41:39.294167 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.294098    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"openshift-service-ca.crt\""
Oct 31 00:41:39.294167 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.294109    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"kube-root-ca.crt\""
Oct 31 00:41:39.368205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.368171    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-home\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.368205 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.368207    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-dshm\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.368422 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.368231    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-kserve-provision-location\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.368422 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.368332    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-model-cache\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.368422 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.368375    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6wgm\" (UniqueName: \"kubernetes.io/projected/922f7b09-40a8-4e83-b938-c9ccccb56104-kube-api-access-l6wgm\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.368511 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.368469    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-tmp-dir\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.368511 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.368502    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922f7b09-40a8-4e83-b938-c9ccccb56104-tls-certs\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.469759 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.469667    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-tmp-dir\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.469759 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.469715    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922f7b09-40a8-4e83-b938-c9ccccb56104-tls-certs\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.469759 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.469758    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-home\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.470065 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.469782    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-dshm\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.470065 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.469824    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-kserve-provision-location\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.470065 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.469872    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-model-cache\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.470065 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.469919    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6wgm\" (UniqueName: \"kubernetes.io/projected/922f7b09-40a8-4e83-b938-c9ccccb56104-kube-api-access-l6wgm\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.470260 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.470205    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-tmp-dir\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.470260 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.470245    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-home\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.470393 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.470315    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-model-cache\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.470454 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.470392    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-kserve-provision-location\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.472330 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.472311    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-dshm\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.472576 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.472558    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922f7b09-40a8-4e83-b938-c9ccccb56104-tls-certs\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.530604 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.530574    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6wgm\" (UniqueName: \"kubernetes.io/projected/922f7b09-40a8-4e83-b938-c9ccccb56104-kube-api-access-l6wgm\") pod \"cpu-accel-preset-kserve-7f685fc657-kw6wp\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:39.810864 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:39.810820    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:41:40.340066 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:40.340014    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"]
Oct 31 00:41:40.346520 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:41:40.346489    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922f7b09_40a8_4e83_b938_c9ccccb56104.slice/crio-7f0c91c3e3b6d4cfcd2039e27f10062e7794b2f6fc6984ce8ca9829f5dd123fa WatchSource:0}: Error finding container 7f0c91c3e3b6d4cfcd2039e27f10062e7794b2f6fc6984ce8ca9829f5dd123fa: Status 404 returned error can't find the container with id 7f0c91c3e3b6d4cfcd2039e27f10062e7794b2f6fc6984ce8ca9829f5dd123fa
Oct 31 00:41:41.346189 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:41.346146    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp" event={"ID":"922f7b09-40a8-4e83-b938-c9ccccb56104","Type":"ContainerStarted","Data":"b502300757581dab33e723edf3f880ba71a020f3e7fe0787bf925144a2b825bf"}
Oct 31 00:41:41.346189 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:41.346194    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp" event={"ID":"922f7b09-40a8-4e83-b938-c9ccccb56104","Type":"ContainerStarted","Data":"7f0c91c3e3b6d4cfcd2039e27f10062e7794b2f6fc6984ce8ca9829f5dd123fa"}
Oct 31 00:41:42.022456 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.022423    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"]
Oct 31 00:41:42.026178 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.026160    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.077725 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.077694    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Oct 31 00:41:42.077913 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.077840    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Oct 31 00:41:42.077913 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.077855    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Oct 31 00:41:42.078117 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.078098    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-7hq4t\""
Oct 31 00:41:42.092429 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.092382    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.092621 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.092447    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfchd\" (UniqueName: \"kubernetes.io/projected/06855c49-837b-4ea2-adb1-8d2c9359dd01-kube-api-access-pfchd\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.092621 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.092576    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.092621 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.092612    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.092799 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.092643    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.092799 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.092676    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06855c49-837b-4ea2-adb1-8d2c9359dd01-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.124718 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.124682    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"]
Oct 31 00:41:42.193489 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.193451    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.193489 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.193492    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.193769 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.193520    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.193769 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.193552    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06855c49-837b-4ea2-adb1-8d2c9359dd01-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.193769 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.193619    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.193769 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.193660    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfchd\" (UniqueName: \"kubernetes.io/projected/06855c49-837b-4ea2-adb1-8d2c9359dd01-kube-api-access-pfchd\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.194077 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.194022    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.194222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.194116    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.194313 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.194292    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.196635 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.196602    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.197120 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.197096    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06855c49-837b-4ea2-adb1-8d2c9359dd01-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.256951 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.256911    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfchd\" (UniqueName: \"kubernetes.io/projected/06855c49-837b-4ea2-adb1-8d2c9359dd01-kube-api-access-pfchd\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-ms94c\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.339750 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.339670    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:42.737519 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.737488    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"]
Oct 31 00:41:42.743419 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:41:42.743388    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06855c49_837b_4ea2_adb1_8d2c9359dd01.slice/crio-d7d8f997099b33912cb853176db0a473171ea2c4c4ef60f57efa0f56bd6e7006 WatchSource:0}: Error finding container d7d8f997099b33912cb853176db0a473171ea2c4c4ef60f57efa0f56bd6e7006: Status 404 returned error can't find the container with id d7d8f997099b33912cb853176db0a473171ea2c4c4ef60f57efa0f56bd6e7006
Oct 31 00:41:42.745652 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:42.745630    4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:41:43.362086 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:43.362028    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c" event={"ID":"06855c49-837b-4ea2-adb1-8d2c9359dd01","Type":"ContainerStarted","Data":"d7d8f997099b33912cb853176db0a473171ea2c4c4ef60f57efa0f56bd6e7006"}
Oct 31 00:41:44.158393 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:41:44.158354    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:41:45.374221 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:45.374191    4935 generic.go:356] "Generic (PLEG): container finished" podID="922f7b09-40a8-4e83-b938-c9ccccb56104" containerID="b502300757581dab33e723edf3f880ba71a020f3e7fe0787bf925144a2b825bf" exitCode=0
Oct 31 00:41:45.374601 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:45.374259    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp" event={"ID":"922f7b09-40a8-4e83-b938-c9ccccb56104","Type":"ContainerDied","Data":"b502300757581dab33e723edf3f880ba71a020f3e7fe0787bf925144a2b825bf"}
Oct 31 00:41:46.380069 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:46.380016    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c" event={"ID":"06855c49-837b-4ea2-adb1-8d2c9359dd01","Type":"ContainerStarted","Data":"79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98"}
Oct 31 00:41:46.493361 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:46.493305    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c" podStartSLOduration=2.919312054 podStartE2EDuration="5.493291205s" totalImagesPullingTime="2.573979151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:46.492380682 +0000 UTC m=+778.674909977" watchObservedRunningTime="2026-10-31 00:41:46.493291205 +0000 UTC m=+778.675820549"
Oct 31 00:41:46.855047 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:46.855002    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"]
Oct 31 00:41:47.339909 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:47.339864    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:52.340167 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:52.340126    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:52.358492 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:52.358458    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:41:52.422362 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:41:52.421668    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:42:14.161497 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:42:14.161447    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:42:24.518577 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.518540    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"]
Oct 31 00:42:24.529207 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:42:24.529164    4935 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98" cmd=["/bin/sleep","15"]
Oct 31 00:42:24.529398 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.529234    4935 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:42:24.529398 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:42:24.529247    4935 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c" podUID="06855c49-837b-4ea2-adb1-8d2c9359dd01" containerName="main" containerID="cri-o://79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98"
Oct 31 00:42:24.529398 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.529289    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c" podUID="06855c49-837b-4ea2-adb1-8d2c9359dd01" containerName="main" containerID="cri-o://79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98" gracePeriod=60
Oct 31 00:42:24.890830 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.890799    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:42:24.996509 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.996455    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-home\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-home\") pod \"06855c49-837b-4ea2-adb1-8d2c9359dd01\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") "
Oct 31 00:42:24.996681 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.996533    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06855c49-837b-4ea2-adb1-8d2c9359dd01-tls-certs\" (UniqueName: \"kubernetes.io/secret/06855c49-837b-4ea2-adb1-8d2c9359dd01-tls-certs\") pod \"06855c49-837b-4ea2-adb1-8d2c9359dd01\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") "
Oct 31 00:42:24.996681 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.996577    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-dshm\") pod \"06855c49-837b-4ea2-adb1-8d2c9359dd01\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") "
Oct 31 00:42:24.996681 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.996613    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-model-cache\") pod \"06855c49-837b-4ea2-adb1-8d2c9359dd01\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") "
Oct 31 00:42:24.996681 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.996648    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06855c49-837b-4ea2-adb1-8d2c9359dd01-kube-api-access-pfchd\" (UniqueName: \"kubernetes.io/projected/06855c49-837b-4ea2-adb1-8d2c9359dd01-kube-api-access-pfchd\") pod \"06855c49-837b-4ea2-adb1-8d2c9359dd01\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") "
Oct 31 00:42:24.996681 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.996673    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-tmp-dir\") pod \"06855c49-837b-4ea2-adb1-8d2c9359dd01\" (UID: \"06855c49-837b-4ea2-adb1-8d2c9359dd01\") "
Oct 31 00:42:24.996971 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.996729    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-home" pod "06855c49-837b-4ea2-adb1-8d2c9359dd01" (UID: "06855c49-837b-4ea2-adb1-8d2c9359dd01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:24.996971 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.996945    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:24.997123 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.996967    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-model-cache" pod "06855c49-837b-4ea2-adb1-8d2c9359dd01" (UID: "06855c49-837b-4ea2-adb1-8d2c9359dd01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:24.997123 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.996979    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-tmp-dir" pod "06855c49-837b-4ea2-adb1-8d2c9359dd01" (UID: "06855c49-837b-4ea2-adb1-8d2c9359dd01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:24.999192 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.999159    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-dshm" pod "06855c49-837b-4ea2-adb1-8d2c9359dd01" (UID: "06855c49-837b-4ea2-adb1-8d2c9359dd01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:24.999338 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.999208    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06855c49-837b-4ea2-adb1-8d2c9359dd01-tls-certs" pod "06855c49-837b-4ea2-adb1-8d2c9359dd01" (UID: "06855c49-837b-4ea2-adb1-8d2c9359dd01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:42:24.999338 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:24.999299    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06855c49-837b-4ea2-adb1-8d2c9359dd01-kube-api-access-pfchd" pod "06855c49-837b-4ea2-adb1-8d2c9359dd01" (UID: "06855c49-837b-4ea2-adb1-8d2c9359dd01"). InnerVolumeSpecName "kube-api-access-pfchd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:25.097760 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.097677    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06855c49-837b-4ea2-adb1-8d2c9359dd01-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:25.097760 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.097708    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:25.097760 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.097717    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:25.097760 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.097732    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfchd\" (UniqueName: \"kubernetes.io/projected/06855c49-837b-4ea2-adb1-8d2c9359dd01-kube-api-access-pfchd\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:25.097760 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.097748    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06855c49-837b-4ea2-adb1-8d2c9359dd01-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:42:25.575919 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.575886    4935 generic.go:356] "Generic (PLEG): container finished" podID="06855c49-837b-4ea2-adb1-8d2c9359dd01" containerID="79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98" exitCode=0
Oct 31 00:42:25.576364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.575933    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c" event={"ID":"06855c49-837b-4ea2-adb1-8d2c9359dd01","Type":"ContainerDied","Data":"79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98"}
Oct 31 00:42:25.576364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.575951    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"
Oct 31 00:42:25.576364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.575976    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c" event={"ID":"06855c49-837b-4ea2-adb1-8d2c9359dd01","Type":"ContainerDied","Data":"d7d8f997099b33912cb853176db0a473171ea2c4c4ef60f57efa0f56bd6e7006"}
Oct 31 00:42:25.576364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.575999    4935 scope.go:122] "RemoveContainer" containerID="79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98"
Oct 31 00:42:25.585954 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.585936    4935 scope.go:122] "RemoveContainer" containerID="79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98"
Oct 31 00:42:25.586310 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:42:25.586287    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98\": container with ID starting with 79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98 not found: ID does not exist" containerID="79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98"
Oct 31 00:42:25.586364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.586319    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98"} err="failed to get container status \"79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98\": rpc error: code = NotFound desc = could not find container \"79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98\": container with ID starting with 79635f37ab9426da7ad3e21eb63575e55ef1ce97bc606a375023a6e23a693a98 not found: ID does not exist"
Oct 31 00:42:25.734236 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.734201    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"]
Oct 31 00:42:25.781191 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.781151    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-ms94c"]
Oct 31 00:42:25.942970 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:25.942891    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06855c49-837b-4ea2-adb1-8d2c9359dd01" path="/var/lib/kubelet/pods/06855c49-837b-4ea2-adb1-8d2c9359dd01/volumes"
Oct 31 00:42:44.157405 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:42:44.157366    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:42:48.024202 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:42:48.024165    4935 scope.go:122] "RemoveContainer" containerID="7fdc1558115d106e5c6157c5aeda1891f6ea2b537455d1013fb42bbe162ed9cf"
Oct 31 00:43:00.755406 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:00.755359    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp" event={"ID":"922f7b09-40a8-4e83-b938-c9ccccb56104","Type":"ContainerStarted","Data":"edc084851ec57a896aa2040de9e372d0034376871ba6e1d37ab680353f7c7121"}
Oct 31 00:43:00.811943 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:00.811905    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:43:00.968547 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:00.968491    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp" podStartSLOduration=2.78114137 podStartE2EDuration="1m21.968475844s" totalImagesPullingTime="1m14.375516693s" totalInitContainerRuntime="4.811817781s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:00.920376975 +0000 UTC m=+853.102906268" watchObservedRunningTime="2026-10-31 00:43:00.968475844 +0000 UTC m=+853.151005146"
Oct 31 00:43:14.153302 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:43:14.153178    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:43:15.775939 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:15.775879    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp" podUID="922f7b09-40a8-4e83-b938-c9ccccb56104" containerName="main" containerID="cri-o://edc084851ec57a896aa2040de9e372d0034376871ba6e1d37ab680353f7c7121" gracePeriod=45
Oct 31 00:43:17.451428 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.451389    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"]
Oct 31 00:43:17.451818 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.451759    4935 cpu_manager.go:414] "containerMap: removing container" podUID="06855c49-837b-4ea2-adb1-8d2c9359dd01" containerName="main"
Oct 31 00:43:17.451818 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.451771    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06855c49-837b-4ea2-adb1-8d2c9359dd01" containerName="main"
Oct 31 00:43:17.451885 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.451835    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="06855c49-837b-4ea2-adb1-8d2c9359dd01" containerName="main"
Oct 31 00:43:17.473331 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.473300    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.511406 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.511370    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"]
Oct 31 00:43:17.555820 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.555790    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Oct 31 00:43:17.572082 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.572023    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-s9fdl\""
Oct 31 00:43:17.572082 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.572058    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Oct 31 00:43:17.572248 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.572097    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Oct 31 00:43:17.579299 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.579271    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.579396 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.579328    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmnh\" (UniqueName: \"kubernetes.io/projected/431ef032-6b5f-4757-8726-d5e6c076f04d-kube-api-access-7gmnh\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.579496 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.579466    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.579625 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.579586    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.579688 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.579620    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431ef032-6b5f-4757-8726-d5e6c076f04d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.579688 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.579655    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.680770 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.680734    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.680770 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.680773    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431ef032-6b5f-4757-8726-d5e6c076f04d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.681029 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.680799    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.681029 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.680916    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.681029 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.680996    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmnh\" (UniqueName: \"kubernetes.io/projected/431ef032-6b5f-4757-8726-d5e6c076f04d-kube-api-access-7gmnh\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.681252 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.681074    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.681252 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.681232    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.681503 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.681466    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.681503 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.681480    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.683356 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.683327    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.683645 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.683626    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431ef032-6b5f-4757-8726-d5e6c076f04d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.741189 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.741151    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmnh\" (UniqueName: \"kubernetes.io/projected/431ef032-6b5f-4757-8726-d5e6c076f04d-kube-api-access-7gmnh\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:17.804382 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:17.804339    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:18.203862 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:18.203775    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"]
Oct 31 00:43:18.210275 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:43:18.210244    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431ef032_6b5f_4757_8726_d5e6c076f04d.slice/crio-d5e1a63265e18c399bb48a7bbe14f6ef90c0cd1b864e6f2f34a71f2fda0f574c WatchSource:0}: Error finding container d5e1a63265e18c399bb48a7bbe14f6ef90c0cd1b864e6f2f34a71f2fda0f574c: Status 404 returned error can't find the container with id d5e1a63265e18c399bb48a7bbe14f6ef90c0cd1b864e6f2f34a71f2fda0f574c
Oct 31 00:43:18.840705 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:18.840669    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm" event={"ID":"431ef032-6b5f-4757-8726-d5e6c076f04d","Type":"ContainerStarted","Data":"c7cd483b5e7fc024a1f00de81ae2b641cfbc9dbef131b6ba9983e05c109a7dc8"}
Oct 31 00:43:18.841197 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:18.840731    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm" event={"ID":"431ef032-6b5f-4757-8726-d5e6c076f04d","Type":"ContainerStarted","Data":"d5e1a63265e18c399bb48a7bbe14f6ef90c0cd1b864e6f2f34a71f2fda0f574c"}
Oct 31 00:43:18.953506 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:18.953453    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm" podStartSLOduration=1.9534395949999999 podStartE2EDuration="1.953439595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:18.952656696 +0000 UTC m=+871.135185990" watchObservedRunningTime="2026-10-31 00:43:18.953439595 +0000 UTC m=+871.135968889"
Oct 31 00:43:19.804755 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:19.804708    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:27.805200 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:27.805161    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:27.817662 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:27.817635    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:27.895013 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:27.894984    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:44.153060 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:43:44.153010    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:43:48.014607 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:48.014579    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:43:48.016765 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:48.016745    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:43:58.802330 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:58.802291    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"]
Oct 31 00:43:58.812278 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:43:58.812233    4935 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="c7cd483b5e7fc024a1f00de81ae2b641cfbc9dbef131b6ba9983e05c109a7dc8" cmd=["/bin/sleep","15"]
Oct 31 00:43:58.812471 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:58.812303    4935 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:43:58.812471 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:43:58.812322    4935 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm" podUID="431ef032-6b5f-4757-8726-d5e6c076f04d" containerName="main" containerID="cri-o://c7cd483b5e7fc024a1f00de81ae2b641cfbc9dbef131b6ba9983e05c109a7dc8"
Oct 31 00:43:58.812471 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:58.812381    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm" podUID="431ef032-6b5f-4757-8726-d5e6c076f04d" containerName="main" containerID="cri-o://c7cd483b5e7fc024a1f00de81ae2b641cfbc9dbef131b6ba9983e05c109a7dc8" gracePeriod=60
Oct 31 00:43:58.883723 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:58.883681    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm" podUID="431ef032-6b5f-4757-8726-d5e6c076f04d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 31 00:43:59.040478 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.040447    4935 generic.go:356] "Generic (PLEG): container finished" podID="431ef032-6b5f-4757-8726-d5e6c076f04d" containerID="c7cd483b5e7fc024a1f00de81ae2b641cfbc9dbef131b6ba9983e05c109a7dc8" exitCode=0
Oct 31 00:43:59.040478 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.040485    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm" event={"ID":"431ef032-6b5f-4757-8726-d5e6c076f04d","Type":"ContainerDied","Data":"c7cd483b5e7fc024a1f00de81ae2b641cfbc9dbef131b6ba9983e05c109a7dc8"}
Oct 31 00:43:59.158869 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.158843    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:43:59.262515 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.262480    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-home\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-home\") pod \"431ef032-6b5f-4757-8726-d5e6c076f04d\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") "
Oct 31 00:43:59.262684 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.262525    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431ef032-6b5f-4757-8726-d5e6c076f04d-kube-api-access-7gmnh\" (UniqueName: \"kubernetes.io/projected/431ef032-6b5f-4757-8726-d5e6c076f04d-kube-api-access-7gmnh\") pod \"431ef032-6b5f-4757-8726-d5e6c076f04d\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") "
Oct 31 00:43:59.262684 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.262569    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-tmp-dir\") pod \"431ef032-6b5f-4757-8726-d5e6c076f04d\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") "
Oct 31 00:43:59.262684 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.262596    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431ef032-6b5f-4757-8726-d5e6c076f04d-tls-certs\" (UniqueName: \"kubernetes.io/secret/431ef032-6b5f-4757-8726-d5e6c076f04d-tls-certs\") pod \"431ef032-6b5f-4757-8726-d5e6c076f04d\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") "
Oct 31 00:43:59.262839 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.262716    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-model-cache\") pod \"431ef032-6b5f-4757-8726-d5e6c076f04d\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") "
Oct 31 00:43:59.262839 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.262753    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-dshm\") pod \"431ef032-6b5f-4757-8726-d5e6c076f04d\" (UID: \"431ef032-6b5f-4757-8726-d5e6c076f04d\") "
Oct 31 00:43:59.262839 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.262821    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-home" pod "431ef032-6b5f-4757-8726-d5e6c076f04d" (UID: "431ef032-6b5f-4757-8726-d5e6c076f04d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:59.262991 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.262876    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-tmp-dir" pod "431ef032-6b5f-4757-8726-d5e6c076f04d" (UID: "431ef032-6b5f-4757-8726-d5e6c076f04d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:59.262991 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.262971    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-model-cache" pod "431ef032-6b5f-4757-8726-d5e6c076f04d" (UID: "431ef032-6b5f-4757-8726-d5e6c076f04d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:59.263127 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.263094    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:43:59.263127 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.263113    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:43:59.263197 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.263128    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:43:59.265018 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.264996    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-dshm" pod "431ef032-6b5f-4757-8726-d5e6c076f04d" (UID: "431ef032-6b5f-4757-8726-d5e6c076f04d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:59.265340 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.265323    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431ef032-6b5f-4757-8726-d5e6c076f04d-kube-api-access-7gmnh" pod "431ef032-6b5f-4757-8726-d5e6c076f04d" (UID: "431ef032-6b5f-4757-8726-d5e6c076f04d"). InnerVolumeSpecName "kube-api-access-7gmnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:59.265340 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.265325    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431ef032-6b5f-4757-8726-d5e6c076f04d-tls-certs" pod "431ef032-6b5f-4757-8726-d5e6c076f04d" (UID: "431ef032-6b5f-4757-8726-d5e6c076f04d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:43:59.364348 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.364267    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431ef032-6b5f-4757-8726-d5e6c076f04d-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:43:59.364348 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.364296    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431ef032-6b5f-4757-8726-d5e6c076f04d-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:43:59.364348 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:43:59.364306    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gmnh\" (UniqueName: \"kubernetes.io/projected/431ef032-6b5f-4757-8726-d5e6c076f04d-kube-api-access-7gmnh\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:00.054217 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:00.054183    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"
Oct 31 00:44:00.054217 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:00.054196    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm" event={"ID":"431ef032-6b5f-4757-8726-d5e6c076f04d","Type":"ContainerDied","Data":"d5e1a63265e18c399bb48a7bbe14f6ef90c0cd1b864e6f2f34a71f2fda0f574c"}
Oct 31 00:44:00.054722 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:00.054310    4935 scope.go:122] "RemoveContainer" containerID="c7cd483b5e7fc024a1f00de81ae2b641cfbc9dbef131b6ba9983e05c109a7dc8"
Oct 31 00:44:00.213144 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:00.213055    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"]
Oct 31 00:44:00.259929 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:00.259895    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76dc6xm"]
Oct 31 00:44:01.060255 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.060181    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-7f685fc657-kw6wp_922f7b09-40a8-4e83-b938-c9ccccb56104/main/0.log"
Oct 31 00:44:01.060675 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.060566    4935 generic.go:356] "Generic (PLEG): container finished" podID="922f7b09-40a8-4e83-b938-c9ccccb56104" containerID="edc084851ec57a896aa2040de9e372d0034376871ba6e1d37ab680353f7c7121" exitCode=137
Oct 31 00:44:01.060675 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.060631    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp" event={"ID":"922f7b09-40a8-4e83-b938-c9ccccb56104","Type":"ContainerDied","Data":"edc084851ec57a896aa2040de9e372d0034376871ba6e1d37ab680353f7c7121"}
Oct 31 00:44:01.422426 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.422405    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-7f685fc657-kw6wp_922f7b09-40a8-4e83-b938-c9ccccb56104/main/0.log"
Oct 31 00:44:01.423758 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.423743    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:44:01.584935 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.584893    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-kserve-provision-location\") pod \"922f7b09-40a8-4e83-b938-c9ccccb56104\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") "
Oct 31 00:44:01.584935 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.584941    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-model-cache\") pod \"922f7b09-40a8-4e83-b938-c9ccccb56104\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") "
Oct 31 00:44:01.585222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.584975    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922f7b09-40a8-4e83-b938-c9ccccb56104-kube-api-access-l6wgm\" (UniqueName: \"kubernetes.io/projected/922f7b09-40a8-4e83-b938-c9ccccb56104-kube-api-access-l6wgm\") pod \"922f7b09-40a8-4e83-b938-c9ccccb56104\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") "
Oct 31 00:44:01.585222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.585083    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/922f7b09-40a8-4e83-b938-c9ccccb56104-tls-certs\" (UniqueName: \"kubernetes.io/secret/922f7b09-40a8-4e83-b938-c9ccccb56104-tls-certs\") pod \"922f7b09-40a8-4e83-b938-c9ccccb56104\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") "
Oct 31 00:44:01.585222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.585131    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-dshm\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-dshm\") pod \"922f7b09-40a8-4e83-b938-c9ccccb56104\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") "
Oct 31 00:44:01.585222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.585171    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-tmp-dir\") pod \"922f7b09-40a8-4e83-b938-c9ccccb56104\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") "
Oct 31 00:44:01.585414 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.585233    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-home\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-home\") pod \"922f7b09-40a8-4e83-b938-c9ccccb56104\" (UID: \"922f7b09-40a8-4e83-b938-c9ccccb56104\") "
Oct 31 00:44:01.585414 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.585252    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-model-cache" pod "922f7b09-40a8-4e83-b938-c9ccccb56104" (UID: "922f7b09-40a8-4e83-b938-c9ccccb56104"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:01.585506 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.585481    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:01.585743 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.585698    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-home" pod "922f7b09-40a8-4e83-b938-c9ccccb56104" (UID: "922f7b09-40a8-4e83-b938-c9ccccb56104"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:01.586329 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.586300    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-tmp-dir" pod "922f7b09-40a8-4e83-b938-c9ccccb56104" (UID: "922f7b09-40a8-4e83-b938-c9ccccb56104"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:01.587978 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.587926    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922f7b09-40a8-4e83-b938-c9ccccb56104-kube-api-access-l6wgm" pod "922f7b09-40a8-4e83-b938-c9ccccb56104" (UID: "922f7b09-40a8-4e83-b938-c9ccccb56104"). InnerVolumeSpecName "kube-api-access-l6wgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:44:01.588091 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.587888    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922f7b09-40a8-4e83-b938-c9ccccb56104-tls-certs" pod "922f7b09-40a8-4e83-b938-c9ccccb56104" (UID: "922f7b09-40a8-4e83-b938-c9ccccb56104"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:44:01.588259 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.588233    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-dshm" pod "922f7b09-40a8-4e83-b938-c9ccccb56104" (UID: "922f7b09-40a8-4e83-b938-c9ccccb56104"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:01.646756 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.646717    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-kserve-provision-location" pod "922f7b09-40a8-4e83-b938-c9ccccb56104" (UID: "922f7b09-40a8-4e83-b938-c9ccccb56104"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:01.686378 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.686336    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:01.686378 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.686372    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6wgm\" (UniqueName: \"kubernetes.io/projected/922f7b09-40a8-4e83-b938-c9ccccb56104-kube-api-access-l6wgm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:01.686378 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.686386    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/922f7b09-40a8-4e83-b938-c9ccccb56104-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:01.686621 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.686398    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:01.686621 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.686436    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:01.686621 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.686448    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/922f7b09-40a8-4e83-b938-c9ccccb56104-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:44:01.943310 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:01.943223    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="431ef032-6b5f-4757-8726-d5e6c076f04d" path="/var/lib/kubelet/pods/431ef032-6b5f-4757-8726-d5e6c076f04d/volumes"
Oct 31 00:44:02.067131 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.067103    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-7f685fc657-kw6wp_922f7b09-40a8-4e83-b938-c9ccccb56104/main/0.log"
Oct 31 00:44:02.067564 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.067484    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp" event={"ID":"922f7b09-40a8-4e83-b938-c9ccccb56104","Type":"ContainerDied","Data":"7f0c91c3e3b6d4cfcd2039e27f10062e7794b2f6fc6984ce8ca9829f5dd123fa"}
Oct 31 00:44:02.067564 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.067512    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"
Oct 31 00:44:02.067564 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.067523    4935 scope.go:122] "RemoveContainer" containerID="edc084851ec57a896aa2040de9e372d0034376871ba6e1d37ab680353f7c7121"
Oct 31 00:44:02.076021 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.076003    4935 scope.go:122] "RemoveContainer" containerID="b502300757581dab33e723edf3f880ba71a020f3e7fe0787bf925144a2b825bf"
Oct 31 00:44:02.120516 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.120487    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"]
Oct 31 00:44:02.120833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.120822    4935 cpu_manager.go:414] "containerMap: removing container" podUID="922f7b09-40a8-4e83-b938-c9ccccb56104" containerName="main"
Oct 31 00:44:02.120873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.120835    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922f7b09-40a8-4e83-b938-c9ccccb56104" containerName="main"
Oct 31 00:44:02.120873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.120860    4935 cpu_manager.go:414] "containerMap: removing container" podUID="922f7b09-40a8-4e83-b938-c9ccccb56104" containerName="storage-initializer"
Oct 31 00:44:02.120873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.120865    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922f7b09-40a8-4e83-b938-c9ccccb56104" containerName="storage-initializer"
Oct 31 00:44:02.120973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.120877    4935 cpu_manager.go:414] "containerMap: removing container" podUID="431ef032-6b5f-4757-8726-d5e6c076f04d" containerName="main"
Oct 31 00:44:02.120973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.120883    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431ef032-6b5f-4757-8726-d5e6c076f04d" containerName="main"
Oct 31 00:44:02.120973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.120948    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="922f7b09-40a8-4e83-b938-c9ccccb56104" containerName="main"
Oct 31 00:44:02.120973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.120960    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="431ef032-6b5f-4757-8726-d5e6c076f04d" containerName="main"
Oct 31 00:44:02.125656 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.125633    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.176664 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.176632    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 00:44:02.176821 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.176707    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-q9j8d\""
Oct 31 00:44:02.220160 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.220078    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"]
Oct 31 00:44:02.228278 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.228236    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"]
Oct 31 00:44:02.266404 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.266372    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7f685fc657-kw6wp"]
Oct 31 00:44:02.291504 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.291467    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.291674 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.291509    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.291674 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.291536    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w496w\" (UniqueName: \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-kube-api-access-w496w\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.291674 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.291650    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.291783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.291724    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.291783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.291752    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.291783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.291779    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.291881 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.291852    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.291924 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.291883    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.392932 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.392884    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.392932 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.392939    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w496w\" (UniqueName: \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-kube-api-access-w496w\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.392960    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.393006    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.393023    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.393091    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.393124    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.393156    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.393205    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393558 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.393477    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393558 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.393516    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393675 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.393653    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393735 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.393710    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.393957 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.393939    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.395872 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.395853    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.396210 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.396191    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.581257 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.581220    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.581443 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.581306    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w496w\" (UniqueName: \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-kube-api-access-w496w\") pod \"openshift-ai-inference-openshift-default-568c495856-h2nt2\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:02.738271 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:02.738231    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:03.165018 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:03.164980    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"]
Oct 31 00:44:03.173734 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:44:03.173708    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f48454a_47d5_4f0b_b215_c64be8f3baa4.slice/crio-df3bdfcf02e17263c00e6ecee840fe0337286a6b15a43380387a4f6cced3f7cc WatchSource:0}: Error finding container df3bdfcf02e17263c00e6ecee840fe0337286a6b15a43380387a4f6cced3f7cc: Status 404 returned error can't find the container with id df3bdfcf02e17263c00e6ecee840fe0337286a6b15a43380387a4f6cced3f7cc
Oct 31 00:44:03.943429 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:03.943393    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922f7b09-40a8-4e83-b938-c9ccccb56104" path="/var/lib/kubelet/pods/922f7b09-40a8-4e83-b938-c9ccccb56104/volumes"
Oct 31 00:44:04.078559 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:04.078526    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" event={"ID":"3f48454a-47d5-4f0b-b215-c64be8f3baa4","Type":"ContainerStarted","Data":"df3bdfcf02e17263c00e6ecee840fe0337286a6b15a43380387a4f6cced3f7cc"}
Oct 31 00:44:07.732123 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:07.732090    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:44:07.732411 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:07.732160    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:44:07.732411 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:07.732190    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:44:08.099464 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:08.099422    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" event={"ID":"3f48454a-47d5-4f0b-b215-c64be8f3baa4","Type":"ContainerStarted","Data":"f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f"}
Oct 31 00:44:08.263021 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:08.262968    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" podStartSLOduration=1.707220588 podStartE2EDuration="6.262954576s" totalImagesPullingTime="4.555733988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:08.262748344 +0000 UTC m=+920.445277638" watchObservedRunningTime="2026-10-31 00:44:08.262954576 +0000 UTC m=+920.445483869"
Oct 31 00:44:08.739220 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:08.739178    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:08.743838 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:08.743813    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:09.103954 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:09.103860    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:09.104833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:09.104812    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:44:14.151924 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:44:14.151883    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:44:17.246704 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.246672    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"]
Oct 31 00:44:17.250228 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.250208    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.293706 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.293677    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"]
Oct 31 00:44:17.296942 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.296926    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.307234 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307211    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.307383 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307239    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.307383 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307260    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.307383 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307277    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.307383 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307294    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.307383 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307346    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.307641 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307504    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjs4x\" (UniqueName: \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-kube-api-access-tjs4x\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.307641 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307545    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.307641 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307608    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.307780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307650    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.307780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307666    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.307780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307680    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q45zf\" (UniqueName: \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-kube-api-access-q45zf\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.307780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307718    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.307780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307734    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.307780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307758    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.307780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307772    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.307994 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307790    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.307994 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.307813    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.340759 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.340728    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"]
Oct 31 00:44:17.381251 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.381220    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"]
Oct 31 00:44:17.408266 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408233    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjs4x\" (UniqueName: \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-kube-api-access-tjs4x\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.408481 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408280    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.408481 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408309    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.408481 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408327    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.408481 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408342    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.408481 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408362    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45zf\" (UniqueName: \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-kube-api-access-q45zf\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.408481 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408407    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.408855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408623    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.408855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408667    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.408855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408686    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.408855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408713    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.408855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408746    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.408855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408796    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.408855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408824    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.408855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408848    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.409486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408875    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.409486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408879    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.409486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408904    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.409486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.408937    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.409486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.409300    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.409486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.409225    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.409783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.409563    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.409783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.409673    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.409783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.409758    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.409939 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.409798    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.409939 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.409837    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.410069 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.410016    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.410149 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.410127    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.411735 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.411708    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.411735 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.411725    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.411855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.411771    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.411892 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.411853    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.470508 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.470479    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.470682 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.470567    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.470682 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.470592    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q45zf\" (UniqueName: \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-kube-api-access-q45zf\") pod \"openshift-ai-inference-openshift-default-568c495856-j4vnw\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.470802 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.470720    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjs4x\" (UniqueName: \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-kube-api-access-tjs4x\") pod \"openshift-ai-inference-openshift-default-568c495856-kq5fz\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.562632 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.562545    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:17.609399 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.609345    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:17.992670 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.992637    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"]
Oct 31 00:44:17.995076 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:17.994892    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"]
Oct 31 00:44:17.999217 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:44:17.999186    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d45ed09_bb56_4f2e_aa5e_1aceab5ce105.slice/crio-4552934c33a3e499e580d4dac7b05d7d491d4ebec92be652e1fdb01a5da358d7 WatchSource:0}: Error finding container 4552934c33a3e499e580d4dac7b05d7d491d4ebec92be652e1fdb01a5da358d7: Status 404 returned error can't find the container with id 4552934c33a3e499e580d4dac7b05d7d491d4ebec92be652e1fdb01a5da358d7
Oct 31 00:44:18.000813 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:44:18.000694    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441a289f_eb6a_4af9_93e7_30ededcbf8f1.slice/crio-97c6c56c4ddcc8f0ff1608d7a78a879df4a318ffb2d0eb9a54f49c2342c2ab4d WatchSource:0}: Error finding container 97c6c56c4ddcc8f0ff1608d7a78a879df4a318ffb2d0eb9a54f49c2342c2ab4d: Status 404 returned error can't find the container with id 97c6c56c4ddcc8f0ff1608d7a78a879df4a318ffb2d0eb9a54f49c2342c2ab4d
Oct 31 00:44:18.002465 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.002428    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:44:18.002551 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.002509    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:44:18.002609 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.002553    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:44:18.003655 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.003628    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:44:18.003710 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.003685    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:44:18.003750 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.003715    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:44:18.145085 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.145018    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" event={"ID":"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105","Type":"ContainerStarted","Data":"dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf"}
Oct 31 00:44:18.145289 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.145120    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" event={"ID":"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105","Type":"ContainerStarted","Data":"4552934c33a3e499e580d4dac7b05d7d491d4ebec92be652e1fdb01a5da358d7"}
Oct 31 00:44:18.147070 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.147009    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" event={"ID":"441a289f-eb6a-4af9-93e7-30ededcbf8f1","Type":"ContainerStarted","Data":"ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c"}
Oct 31 00:44:18.147216 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.147087    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" event={"ID":"441a289f-eb6a-4af9-93e7-30ededcbf8f1","Type":"ContainerStarted","Data":"97c6c56c4ddcc8f0ff1608d7a78a879df4a318ffb2d0eb9a54f49c2342c2ab4d"}
Oct 31 00:44:18.262634 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.262512    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" podStartSLOduration=1.262492709 podStartE2EDuration="1.262492709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:18.260739907 +0000 UTC m=+930.443269225" watchObservedRunningTime="2026-10-31 00:44:18.262492709 +0000 UTC m=+930.445022004"
Oct 31 00:44:18.376806 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.376743    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" podStartSLOduration=1.376725158 podStartE2EDuration="1.376725158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:18.376541408 +0000 UTC m=+930.559070701" watchObservedRunningTime="2026-10-31 00:44:18.376725158 +0000 UTC m=+930.559254454"
Oct 31 00:44:18.563574 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.563486    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:18.609532 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:18.609498    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:19.568536 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:19.568503    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:19.614643 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:19.614613    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:20.157283 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:20.157246    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:20.157493 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:20.157310    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:20.158062 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:20.158018    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:44:20.158219 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:20.158206    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:44:44.152877 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:44:44.152829    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:44:54.980063 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:54.980014    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"]
Oct 31 00:44:54.984799 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:54.984781    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.035918 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.035882    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Oct 31 00:44:55.082164 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.082135    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Oct 31 00:44:55.082334 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.082189    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-kcf25\""
Oct 31 00:44:55.082334 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.082189    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Oct 31 00:44:55.084241 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.084224    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"]
Oct 31 00:44:55.153578 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.153541    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.153578 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.153577    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqm5\" (UniqueName: \"kubernetes.io/projected/5a713c69-fc0c-499b-ace4-f6cef2ef2221-kube-api-access-jpqm5\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.153826 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.153597    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.153826 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.153728    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.153826 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.153768    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.153982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.153887    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.255167 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.255134    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.255369 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.255199    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.255369 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.255217    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqm5\" (UniqueName: \"kubernetes.io/projected/5a713c69-fc0c-499b-ace4-f6cef2ef2221-kube-api-access-jpqm5\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.255369 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.255333    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.255538 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.255411    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.255538 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.255471    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.255648 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.255624    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.255692 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.255646    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.255692 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.255678    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.257866 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.257837    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.258121 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.258105    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.316985 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.316950    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqm5\" (UniqueName: \"kubernetes.io/projected/5a713c69-fc0c-499b-ace4-f6cef2ef2221-kube-api-access-jpqm5\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-579wm\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.597239 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.597147    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:44:55.973912 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:55.973833    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"]
Oct 31 00:44:55.979654 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:44:55.979621    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a713c69_fc0c_499b_ace4_f6cef2ef2221.slice/crio-5d72747943ca5beb1951a8bab9b15b68535c59ed06c16ef9331c4624935f030a WatchSource:0}: Error finding container 5d72747943ca5beb1951a8bab9b15b68535c59ed06c16ef9331c4624935f030a: Status 404 returned error can't find the container with id 5d72747943ca5beb1951a8bab9b15b68535c59ed06c16ef9331c4624935f030a
Oct 31 00:44:56.340060 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:56.340011    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm" event={"ID":"5a713c69-fc0c-499b-ace4-f6cef2ef2221","Type":"ContainerStarted","Data":"e8508c3d05f305dab5c127f85021da857c8832880c3d4825ea9ce50e44d01585"}
Oct 31 00:44:56.340512 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:56.340106    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm" event={"ID":"5a713c69-fc0c-499b-ace4-f6cef2ef2221","Type":"ContainerStarted","Data":"5d72747943ca5beb1951a8bab9b15b68535c59ed06c16ef9331c4624935f030a"}
Oct 31 00:44:56.454366 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:56.454299    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm" podStartSLOduration=2.454281611 podStartE2EDuration="2.454281611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:56.453368765 +0000 UTC m=+968.635898059" watchObservedRunningTime="2026-10-31 00:44:56.454281611 +0000 UTC m=+968.636810908"
Oct 31 00:44:56.597345 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:44:56.597247    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:45:05.597911 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:05.597868    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:45:05.612146 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:05.612107    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:45:06.198257 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:06.198210    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kq5fz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:45:06.198473 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:06.198322    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:06.199146 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:06.199124    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-j4vnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:45:06.199257 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:06.199172    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:06.470969 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:06.470897    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:45:09.403844 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.403810    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-h2nt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": read tcp 10.132.0.2:43700->10.132.0.47:15021: read: connection reset by peer" start-of-body=
Oct 31 00:45:09.404388 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.403880    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": read tcp 10.132.0.2:43700->10.132.0.47:15021: read: connection reset by peer"
Oct 31 00:45:09.474168 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.474136    4935 generic.go:356] "Generic (PLEG): container finished" podID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerID="f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f" exitCode=0
Oct 31 00:45:09.474336 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.474210    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" event={"ID":"3f48454a-47d5-4f0b-b215-c64be8f3baa4","Type":"ContainerDied","Data":"f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f"}
Oct 31 00:45:09.474612 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.474589    4935 scope.go:122] "RemoveContainer" containerID="f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f"
Oct 31 00:45:09.475772 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.475729    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:45:09.475868 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.475813    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:45:09.475868 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.475818    4935 generic.go:356] "Generic (PLEG): container finished" podID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerID="dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf" exitCode=0
Oct 31 00:45:09.475868 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.475860    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" event={"ID":"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105","Type":"ContainerDied","Data":"dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf"}
Oct 31 00:45:09.475868 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.475862    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:45:09.476175 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.476161    4935 scope.go:122] "RemoveContainer" containerID="dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf"
Oct 31 00:45:09.477084 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.477058    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:45:09.477194 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.477129    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:45:09.477194 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:09.477179    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:45:10.503949 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.502146    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" event={"ID":"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105","Type":"ContainerStarted","Data":"b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654"}
Oct 31 00:45:10.506460 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.506425    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" event={"ID":"3f48454a-47d5-4f0b-b215-c64be8f3baa4","Type":"ContainerStarted","Data":"913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c"}
Oct 31 00:45:10.508238 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.508212    4935 generic.go:356] "Generic (PLEG): container finished" podID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerID="ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c" exitCode=0
Oct 31 00:45:10.508354 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.508285    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" event={"ID":"441a289f-eb6a-4af9-93e7-30ededcbf8f1","Type":"ContainerDied","Data":"ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c"}
Oct 31 00:45:10.508605 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.508592    4935 scope.go:122] "RemoveContainer" containerID="ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c"
Oct 31 00:45:10.509693 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.509648    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:45:10.509789 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.509730    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:45:10.509789 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.509773    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:45:10.573061 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.565592    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:45:10.596071 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.587566    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kq5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:45:10.596071 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.587625    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:45:10.739336 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.739304    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:45:10.739665 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.739641    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-h2nt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 31 00:45:10.739796 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:10.739780    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 31 00:45:11.538492 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:11.538459    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" event={"ID":"441a289f-eb6a-4af9-93e7-30ededcbf8f1","Type":"ContainerStarted","Data":"27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d"}
Oct 31 00:45:11.567083 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:11.563438    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kq5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:45:11.567083 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:11.563500    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:45:11.611284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:11.611237    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:45:11.611602 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:11.611460    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-j4vnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 31 00:45:11.611785 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:11.611762    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 31 00:45:11.739455 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:11.739419    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-h2nt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 31 00:45:11.739636 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:11.739477    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 31 00:45:12.571061 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:12.570525    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kq5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:45:12.571061 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:12.570590    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:45:12.612497 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:12.612458    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-j4vnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 31 00:45:12.612808 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:12.612774    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 31 00:45:12.739202 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:12.739170    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-h2nt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 31 00:45:12.739415 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:12.739396    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 31 00:45:13.571565 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:13.571529    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kq5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:45:13.572213 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:13.572180    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:45:13.611478 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:13.611438    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-j4vnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 31 00:45:13.611819 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:13.611790    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 31 00:45:13.738788 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:13.738758    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-h2nt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 31 00:45:13.739025 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:13.739003    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 31 00:45:14.226521 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:45:14.226448    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:45:14.564613 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:14.564578    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kq5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:45:14.565330 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:14.564893    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:45:14.612641 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:14.612517    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-j4vnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 31 00:45:14.612641 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:14.612579    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 31 00:45:14.739161 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:14.739123    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-h2nt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 31 00:45:14.739352 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:14.739184    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 31 00:45:15.566574 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:15.566532    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kq5fz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:45:15.566950 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:15.566917    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:45:15.610512 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:15.610473    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-j4vnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 31 00:45:15.610702 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:15.610542    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 31 00:45:15.742890 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:15.742848    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:45:15.743394 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:15.743272    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:45:15.743850 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:15.743828    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:45:16.567444 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:16.567406    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:45:16.567708 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:16.567686    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:45:16.568513 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:16.568492    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:45:16.610442 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:16.610393    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-j4vnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 31 00:45:16.610645 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:16.610453    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 31 00:45:17.614628 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:17.614597    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:45:17.615124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:17.614855    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:45:17.615684 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:17.615667    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:45:35.439393 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.439355    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"]
Oct 31 00:45:35.448899 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:45:35.448855    4935 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="e8508c3d05f305dab5c127f85021da857c8832880c3d4825ea9ce50e44d01585" cmd=["/bin/sleep","15"]
Oct 31 00:45:35.449078 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.448925    4935 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:45:35.449078 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:45:35.448939    4935 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm" podUID="5a713c69-fc0c-499b-ace4-f6cef2ef2221" containerName="main" containerID="cri-o://e8508c3d05f305dab5c127f85021da857c8832880c3d4825ea9ce50e44d01585"
Oct 31 00:45:35.449078 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.448988    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm" podUID="5a713c69-fc0c-499b-ace4-f6cef2ef2221" containerName="main" containerID="cri-o://e8508c3d05f305dab5c127f85021da857c8832880c3d4825ea9ce50e44d01585" gracePeriod=60
Oct 31 00:45:35.457783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.457738    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm" podUID="5a713c69-fc0c-499b-ace4-f6cef2ef2221" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 31 00:45:35.676253 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.676221    4935 generic.go:356] "Generic (PLEG): container finished" podID="5a713c69-fc0c-499b-ace4-f6cef2ef2221" containerID="e8508c3d05f305dab5c127f85021da857c8832880c3d4825ea9ce50e44d01585" exitCode=0
Oct 31 00:45:35.676417 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.676294    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm" event={"ID":"5a713c69-fc0c-499b-ace4-f6cef2ef2221","Type":"ContainerDied","Data":"e8508c3d05f305dab5c127f85021da857c8832880c3d4825ea9ce50e44d01585"}
Oct 31 00:45:35.797098 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.797071    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:45:35.884786 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.884745    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-dshm\") pod \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") "
Oct 31 00:45:35.885007 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.884818    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tls-certs\") pod \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") "
Oct 31 00:45:35.885007 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.884871    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-home\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-home\") pod \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") "
Oct 31 00:45:35.885007 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.884936    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tmp-dir\") pod \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") "
Oct 31 00:45:35.885007 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.884962    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a713c69-fc0c-499b-ace4-f6cef2ef2221-kube-api-access-jpqm5\" (UniqueName: \"kubernetes.io/projected/5a713c69-fc0c-499b-ace4-f6cef2ef2221-kube-api-access-jpqm5\") pod \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") "
Oct 31 00:45:35.885243 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.885027    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-model-cache\") pod \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\" (UID: \"5a713c69-fc0c-499b-ace4-f6cef2ef2221\") "
Oct 31 00:45:35.885243 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.885221    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tmp-dir" pod "5a713c69-fc0c-499b-ace4-f6cef2ef2221" (UID: "5a713c69-fc0c-499b-ace4-f6cef2ef2221"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:35.885330 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.885236    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-home" pod "5a713c69-fc0c-499b-ace4-f6cef2ef2221" (UID: "5a713c69-fc0c-499b-ace4-f6cef2ef2221"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:35.885420 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.885395    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-model-cache" pod "5a713c69-fc0c-499b-ace4-f6cef2ef2221" (UID: "5a713c69-fc0c-499b-ace4-f6cef2ef2221"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:35.885469 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.885424    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:35.885469 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.885439    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:35.887355 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.887327    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a713c69-fc0c-499b-ace4-f6cef2ef2221-kube-api-access-jpqm5" pod "5a713c69-fc0c-499b-ace4-f6cef2ef2221" (UID: "5a713c69-fc0c-499b-ace4-f6cef2ef2221"). InnerVolumeSpecName "kube-api-access-jpqm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:35.887567 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.887544    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-dshm" pod "5a713c69-fc0c-499b-ace4-f6cef2ef2221" (UID: "5a713c69-fc0c-499b-ace4-f6cef2ef2221"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:35.887567 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.887556    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tls-certs" pod "5a713c69-fc0c-499b-ace4-f6cef2ef2221" (UID: "5a713c69-fc0c-499b-ace4-f6cef2ef2221"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:45:35.986302 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.986030    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpqm5\" (UniqueName: \"kubernetes.io/projected/5a713c69-fc0c-499b-ace4-f6cef2ef2221-kube-api-access-jpqm5\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:35.986302 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.986099    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:35.986302 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.986114    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a713c69-fc0c-499b-ace4-f6cef2ef2221-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:35.986302 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:35.986128    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a713c69-fc0c-499b-ace4-f6cef2ef2221-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:36.681872 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:36.681838    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"
Oct 31 00:45:36.682400 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:36.681842    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm" event={"ID":"5a713c69-fc0c-499b-ace4-f6cef2ef2221","Type":"ContainerDied","Data":"5d72747943ca5beb1951a8bab9b15b68535c59ed06c16ef9331c4624935f030a"}
Oct 31 00:45:36.682400 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:36.681969    4935 scope.go:122] "RemoveContainer" containerID="e8508c3d05f305dab5c127f85021da857c8832880c3d4825ea9ce50e44d01585"
Oct 31 00:45:36.841353 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:36.841318    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"]
Oct 31 00:45:36.888398 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:36.888365    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-579wm"]
Oct 31 00:45:37.943053 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:37.943015    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a713c69-fc0c-499b-ace4-f6cef2ef2221" path="/var/lib/kubelet/pods/5a713c69-fc0c-499b-ace4-f6cef2ef2221/volumes"
Oct 31 00:45:44.153841 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:45:44.153802    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:45:48.747925 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.747890    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"]
Oct 31 00:45:48.748350 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.748281    4935 cpu_manager.go:414] "containerMap: removing container" podUID="5a713c69-fc0c-499b-ace4-f6cef2ef2221" containerName="main"
Oct 31 00:45:48.748350 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.748293    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a713c69-fc0c-499b-ace4-f6cef2ef2221" containerName="main"
Oct 31 00:45:48.748422 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.748374    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a713c69-fc0c-499b-ace4-f6cef2ef2221" containerName="main"
Oct 31 00:45:48.753156 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.753139    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.803542 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.803501    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.803542 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.803540    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.803772 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.803567    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.803772 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.803590    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.803772 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.803676    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.803929 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.803811    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.803929 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.803861    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flk5m\" (UniqueName: \"kubernetes.io/projected/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kube-api-access-flk5m\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.804018 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.803949    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.804715 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.804694    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"seaweedfs-s3-creds\""
Oct 31 00:45:48.850843 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.850811    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-64tdh\""
Oct 31 00:45:48.851059 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.850860    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 00:45:48.851059 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.850812    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Oct 31 00:45:48.851059 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.850831    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Oct 31 00:45:48.851059 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.850880    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Oct 31 00:45:48.852871 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.852848    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"]
Oct 31 00:45:48.905245 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905207    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.905419 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905248    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flk5m\" (UniqueName: \"kubernetes.io/projected/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kube-api-access-flk5m\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.905419 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905290    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.905419 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905383    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.905419 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905407    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.905624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905432    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.905624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905455    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.905624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905480    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.905820 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905773    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.905880 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905831    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.905981 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905949    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.905981 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905972    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.906171 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.905980    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.907926 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.907896    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.908103 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.908084    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:48.965326 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:48.965300    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flk5m\" (UniqueName: \"kubernetes.io/projected/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kube-api-access-flk5m\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:49.067554 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:49.067454    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:49.498911 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:49.498872    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"]
Oct 31 00:45:49.505297 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:45:49.505269    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd853de7f_2ae4_4f15_8a78_cb56c7d2eb90.slice/crio-1256870aa2dd23cf08fdf919bbc90231898ba6e8c3731d1fbf9c092c416b026e WatchSource:0}: Error finding container 1256870aa2dd23cf08fdf919bbc90231898ba6e8c3731d1fbf9c092c416b026e: Status 404 returned error can't find the container with id 1256870aa2dd23cf08fdf919bbc90231898ba6e8c3731d1fbf9c092c416b026e
Oct 31 00:45:49.742203 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:49.742165    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" event={"ID":"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90","Type":"ContainerStarted","Data":"d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f"}
Oct 31 00:45:49.742417 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:49.742221    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" event={"ID":"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90","Type":"ContainerStarted","Data":"1256870aa2dd23cf08fdf919bbc90231898ba6e8c3731d1fbf9c092c416b026e"}
Oct 31 00:45:51.752946 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:51.752903    4935 generic.go:356] "Generic (PLEG): container finished" podID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerID="d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f" exitCode=0
Oct 31 00:45:51.753422 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:51.752976    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" event={"ID":"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90","Type":"ContainerDied","Data":"d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f"}
Oct 31 00:45:52.759789 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:52.759746    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" event={"ID":"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90","Type":"ContainerStarted","Data":"ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984"}
Oct 31 00:45:52.971315 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:52.971264    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" podStartSLOduration=2.870996561 podStartE2EDuration="4.971250392s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.100253831s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:52.923404863 +0000 UTC m=+1025.105934158" watchObservedRunningTime="2026-10-31 00:45:52.971250392 +0000 UTC m=+1025.153779716"
Oct 31 00:45:53.067905 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:53.067814    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:59.067786 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:59.067685    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:45:59.069614 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:45:59.069589    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 00:46:09.068338 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:46:09.068296    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 00:46:14.153672 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:46:14.153627    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:46:19.068405 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:46:19.068361    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 00:46:29.068673 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:46:29.068629    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 00:46:39.068474 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:46:39.068430    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 00:46:44.155464 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:46:44.155423    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:46:49.067934 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:46:49.067879    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 00:46:59.068802 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:46:59.068745    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 00:47:09.077657 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:09.077626    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:47:09.084585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:09.084563    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:47:14.152390 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:47:14.152348    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:47:20.379554 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:20.379512    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"]
Oct 31 00:47:35.399909 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:35.399802    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main" containerID="cri-o://ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984" gracePeriod=45
Oct 31 00:47:36.078658 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.078422    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Oct 31 00:47:36.194190 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.194163    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:47:36.265543 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.265462    4935 generic.go:356] "Generic (PLEG): container finished" podID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerID="ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984" exitCode=0
Oct 31 00:47:36.265543 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.265514    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"
Oct 31 00:47:36.265729 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.265505    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" event={"ID":"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90","Type":"ContainerDied","Data":"ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984"}
Oct 31 00:47:36.265729 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.265633    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp" event={"ID":"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90","Type":"ContainerDied","Data":"1256870aa2dd23cf08fdf919bbc90231898ba6e8c3731d1fbf9c092c416b026e"}
Oct 31 00:47:36.265729 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.265656    4935 scope.go:122] "RemoveContainer" containerID="ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984"
Oct 31 00:47:36.274434 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.274413    4935 scope.go:122] "RemoveContainer" containerID="d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f"
Oct 31 00:47:36.282585 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.282566    4935 scope.go:122] "RemoveContainer" containerID="ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984"
Oct 31 00:47:36.282881 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:47:36.282858    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984\": container with ID starting with ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984 not found: ID does not exist" containerID="ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984"
Oct 31 00:47:36.282924 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.282895    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984"} err="failed to get container status \"ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984\": rpc error: code = NotFound desc = could not find container \"ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984\": container with ID starting with ef99697e4d4db6f06903e3f58cf22cbe654e07121783493727a7ada3a2268984 not found: ID does not exist"
Oct 31 00:47:36.282924 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.282917    4935 scope.go:122] "RemoveContainer" containerID="d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f"
Oct 31 00:47:36.283178 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:47:36.283162    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f\": container with ID starting with d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f not found: ID does not exist" containerID="d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f"
Oct 31 00:47:36.283238 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.283183    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f"} err="failed to get container status \"d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f\": rpc error: code = NotFound desc = could not find container \"d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f\": container with ID starting with d2d5d94fe96eee019f390701a26b3ee8edadb72eea2ca8d88554abad9e680a7f not found: ID does not exist"
Oct 31 00:47:36.289480 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.289455    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-home\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-home\") pod \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") "
Oct 31 00:47:36.289579 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.289519    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tls-certs\" (UniqueName: \"kubernetes.io/secret/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tls-certs\") pod \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") "
Oct 31 00:47:36.289579 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.289546    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tmp-dir\") pod \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") "
Oct 31 00:47:36.289579 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.289574    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-model-cache\") pod \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") "
Oct 31 00:47:36.289729 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.289607    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kserve-provision-location\") pod \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") "
Oct 31 00:47:36.289729 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.289672    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-cabundle-cert\") pod \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") "
Oct 31 00:47:36.289729 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.289713    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kube-api-access-flk5m\" (UniqueName: \"kubernetes.io/projected/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kube-api-access-flk5m\") pod \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") "
Oct 31 00:47:36.289866 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.289830    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-dshm\") pod \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\" (UID: \"d853de7f-2ae4-4f15-8a78-cb56c7d2eb90\") "
Oct 31 00:47:36.289931 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.289905    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-model-cache" pod "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" (UID: "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:47:36.290003 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.289984    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-home" pod "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" (UID: "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:47:36.290195 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.290178    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:36.290271 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.290200    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:36.290642 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.290602    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-cabundle-cert" pod "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" (UID: "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:47:36.292227 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.292197    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tls-certs" pod "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" (UID: "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:47:36.292361 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.292339    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-dshm" pod "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" (UID: "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:47:36.292631 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.292610    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kube-api-access-flk5m" pod "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" (UID: "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90"). InnerVolumeSpecName "kube-api-access-flk5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:47:36.306688 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.306638    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tmp-dir" pod "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" (UID: "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:47:36.316400 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.316374    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kserve-provision-location" pod "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" (UID: "d853de7f-2ae4-4f15-8a78-cb56c7d2eb90"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:47:36.390804 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.390776    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:36.390804 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.390803    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:36.390804 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.390811    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:36.391006 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.390821    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:36.391006 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.390832    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:36.391006 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.390842    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flk5m\" (UniqueName: \"kubernetes.io/projected/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90-kube-api-access-flk5m\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:36.683074 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.683030    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"]
Oct 31 00:47:36.689659 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:36.689637    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7b89f7f65-7h9lp"]
Oct 31 00:47:37.942890 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:47:37.942858    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" path="/var/lib/kubelet/pods/d853de7f-2ae4-4f15-8a78-cb56c7d2eb90/volumes"
Oct 31 00:47:44.151784 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:47:44.151751    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:48:04.801824 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.801785    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"]
Oct 31 00:48:04.802247 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.802197    4935 cpu_manager.go:414] "containerMap: removing container" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="storage-initializer"
Oct 31 00:48:04.802247 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.802212    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="storage-initializer"
Oct 31 00:48:04.802247 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.802224    4935 cpu_manager.go:414] "containerMap: removing container" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main"
Oct 31 00:48:04.802247 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.802229    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main"
Oct 31 00:48:04.802371 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.802314    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d853de7f-2ae4-4f15-8a78-cb56c7d2eb90" containerName="main"
Oct 31 00:48:04.805265 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.805244    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:04.902593 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.902562    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Oct 31 00:48:04.902593 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.902580    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Oct 31 00:48:04.902593 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.902584    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-ptfl2\""
Oct 31 00:48:04.902833 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.902632    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Oct 31 00:48:04.904683 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.904661    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"]
Oct 31 00:48:04.940399 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.940372    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:04.940536 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.940405    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dg9s\" (UniqueName: \"kubernetes.io/projected/22711c8e-a469-44f7-9aae-b900768091e4-kube-api-access-2dg9s\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:04.940536 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.940433    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22711c8e-a469-44f7-9aae-b900768091e4-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:04.940536 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.940516    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:04.940654 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.940538    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:04.940654 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:04.940604    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.041811 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.041775    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.042003 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.041827    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.042003 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.041861    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dg9s\" (UniqueName: \"kubernetes.io/projected/22711c8e-a469-44f7-9aae-b900768091e4-kube-api-access-2dg9s\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.042003 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.041902    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22711c8e-a469-44f7-9aae-b900768091e4-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.042003 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.041940    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.042003 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.041974    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.042316 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.042269    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.042316 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.042300    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.042415 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.042380    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.044374 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.044353    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.044530 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.044513    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22711c8e-a469-44f7-9aae-b900768091e4-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.102447 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.102373    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dg9s\" (UniqueName: \"kubernetes.io/projected/22711c8e-a469-44f7-9aae-b900768091e4-kube-api-access-2dg9s\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-tjf2v\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.117297 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.117267    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:05.490477 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:05.490446    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"]
Oct 31 00:48:05.496655 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:48:05.496626    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22711c8e_a469_44f7_9aae_b900768091e4.slice/crio-adc43f980629fc6ca8971657f0d0977ebe77182b9d7c7b2b1cee131bcae955ad WatchSource:0}: Error finding container adc43f980629fc6ca8971657f0d0977ebe77182b9d7c7b2b1cee131bcae955ad: Status 404 returned error can't find the container with id adc43f980629fc6ca8971657f0d0977ebe77182b9d7c7b2b1cee131bcae955ad
Oct 31 00:48:06.402455 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:06.402416    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v" event={"ID":"22711c8e-a469-44f7-9aae-b900768091e4","Type":"ContainerStarted","Data":"a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d"}
Oct 31 00:48:06.402983 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:06.402473    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v" event={"ID":"22711c8e-a469-44f7-9aae-b900768091e4","Type":"ContainerStarted","Data":"adc43f980629fc6ca8971657f0d0977ebe77182b9d7c7b2b1cee131bcae955ad"}
Oct 31 00:48:06.516553 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:06.516501    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v" podStartSLOduration=2.51648667 podStartE2EDuration="2.51648667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:06.51536306 +0000 UTC m=+1158.697892352" watchObservedRunningTime="2026-10-31 00:48:06.51648667 +0000 UTC m=+1158.699015963"
Oct 31 00:48:07.118174 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:07.118140    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:14.181278 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:48:14.181237    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:48:15.117969 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:15.117928    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:15.129954 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:15.129919    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:15.465737 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:15.465656    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:43.636234 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:43.636198    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"]
Oct 31 00:48:43.646644 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:48:43.646604    4935 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d" cmd=["/bin/sleep","15"]
Oct 31 00:48:43.646818 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:43.646669    4935 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:48:43.646818 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:48:43.646686    4935 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v" podUID="22711c8e-a469-44f7-9aae-b900768091e4" containerName="main" containerID="cri-o://a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d"
Oct 31 00:48:43.646818 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:43.646743    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v" podUID="22711c8e-a469-44f7-9aae-b900768091e4" containerName="main" containerID="cri-o://a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d" gracePeriod=60
Oct 31 00:48:43.997913 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:43.997889    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:44.013130 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.013110    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-dshm\") pod \"22711c8e-a469-44f7-9aae-b900768091e4\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") "
Oct 31 00:48:44.013275 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.013157    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-home\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-home\") pod \"22711c8e-a469-44f7-9aae-b900768091e4\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") "
Oct 31 00:48:44.013275 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.013179    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-tmp-dir\") pod \"22711c8e-a469-44f7-9aae-b900768091e4\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") "
Oct 31 00:48:44.013275 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.013207    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-model-cache\") pod \"22711c8e-a469-44f7-9aae-b900768091e4\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") "
Oct 31 00:48:44.013275 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.013243    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22711c8e-a469-44f7-9aae-b900768091e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/22711c8e-a469-44f7-9aae-b900768091e4-tls-certs\") pod \"22711c8e-a469-44f7-9aae-b900768091e4\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") "
Oct 31 00:48:44.013275 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.013263    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22711c8e-a469-44f7-9aae-b900768091e4-kube-api-access-2dg9s\" (UniqueName: \"kubernetes.io/projected/22711c8e-a469-44f7-9aae-b900768091e4-kube-api-access-2dg9s\") pod \"22711c8e-a469-44f7-9aae-b900768091e4\" (UID: \"22711c8e-a469-44f7-9aae-b900768091e4\") "
Oct 31 00:48:44.013543 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.013406    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-home" pod "22711c8e-a469-44f7-9aae-b900768091e4" (UID: "22711c8e-a469-44f7-9aae-b900768091e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:44.013543 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.013484    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-model-cache" pod "22711c8e-a469-44f7-9aae-b900768091e4" (UID: "22711c8e-a469-44f7-9aae-b900768091e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:44.013543 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.013514    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-tmp-dir" pod "22711c8e-a469-44f7-9aae-b900768091e4" (UID: "22711c8e-a469-44f7-9aae-b900768091e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:44.013543 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.013530    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:44.015405 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.015373    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-dshm" pod "22711c8e-a469-44f7-9aae-b900768091e4" (UID: "22711c8e-a469-44f7-9aae-b900768091e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:48:44.015545 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.015530    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22711c8e-a469-44f7-9aae-b900768091e4-tls-certs" pod "22711c8e-a469-44f7-9aae-b900768091e4" (UID: "22711c8e-a469-44f7-9aae-b900768091e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:48:44.015594 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.015575    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22711c8e-a469-44f7-9aae-b900768091e4-kube-api-access-2dg9s" pod "22711c8e-a469-44f7-9aae-b900768091e4" (UID: "22711c8e-a469-44f7-9aae-b900768091e4"). InnerVolumeSpecName "kube-api-access-2dg9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:44.114973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.114940    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22711c8e-a469-44f7-9aae-b900768091e4-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:44.114973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.114967    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dg9s\" (UniqueName: \"kubernetes.io/projected/22711c8e-a469-44f7-9aae-b900768091e4-kube-api-access-2dg9s\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:44.114973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.114978    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:44.115206 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.114987    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:44.115206 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.114995    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22711c8e-a469-44f7-9aae-b900768091e4-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:44.152186 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:48:44.152148    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:48:44.589924 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.589884    4935 generic.go:356] "Generic (PLEG): container finished" podID="22711c8e-a469-44f7-9aae-b900768091e4" containerID="a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d" exitCode=0
Oct 31 00:48:44.590147 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.589937    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"
Oct 31 00:48:44.590147 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.589965    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v" event={"ID":"22711c8e-a469-44f7-9aae-b900768091e4","Type":"ContainerDied","Data":"a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d"}
Oct 31 00:48:44.590147 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.590020    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v" event={"ID":"22711c8e-a469-44f7-9aae-b900768091e4","Type":"ContainerDied","Data":"adc43f980629fc6ca8971657f0d0977ebe77182b9d7c7b2b1cee131bcae955ad"}
Oct 31 00:48:44.590147 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.590062    4935 scope.go:122] "RemoveContainer" containerID="a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d"
Oct 31 00:48:44.599826 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.599808    4935 scope.go:122] "RemoveContainer" containerID="a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d"
Oct 31 00:48:44.600188 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:48:44.600168    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d\": container with ID starting with a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d not found: ID does not exist" containerID="a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d"
Oct 31 00:48:44.600261 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.600199    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d"} err="failed to get container status \"a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d\": rpc error: code = NotFound desc = could not find container \"a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d\": container with ID starting with a65828b39eedc2318e3c809e96ad0dae6cc8faa1f32547f6afa7bda0ff1d587d not found: ID does not exist"
Oct 31 00:48:44.748496 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.748465    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"]
Oct 31 00:48:44.795412 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:44.795370    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-tjf2v"]
Oct 31 00:48:45.943415 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:45.943381    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22711c8e-a469-44f7-9aae-b900768091e4" path="/var/lib/kubelet/pods/22711c8e-a469-44f7-9aae-b900768091e4/volumes"
Oct 31 00:48:48.050390 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:48.050359    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:48:48.056569 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:48:48.056545    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:49:14.151585 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:49:14.151549    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:49:35.979451 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:35.979419    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"]
Oct 31 00:49:35.979849 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:35.979775    4935 cpu_manager.go:414] "containerMap: removing container" podUID="22711c8e-a469-44f7-9aae-b900768091e4" containerName="main"
Oct 31 00:49:35.979849 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:35.979785    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22711c8e-a469-44f7-9aae-b900768091e4" containerName="main"
Oct 31 00:49:35.979919 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:35.979861    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="22711c8e-a469-44f7-9aae-b900768091e4" containerName="main"
Oct 31 00:49:35.982450 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:35.982430    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.056139 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.056101    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.056318 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.056153    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.056318 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.056267    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6mpm\" (UniqueName: \"kubernetes.io/projected/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-kube-api-access-c6mpm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.056318 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.056307    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.056479 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.056350    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.056479 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.056379    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.082220 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.081527    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Oct 31 00:49:36.082220 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.081969    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-xlmwv\""
Oct 31 00:49:36.082220 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.082077    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-xlkjf\""
Oct 31 00:49:36.082220 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.082153    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Oct 31 00:49:36.082220 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.082154    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Oct 31 00:49:36.083754 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.083731    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"]
Oct 31 00:49:36.156842 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.156809    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.156842 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.156846    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.157119 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.156869    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6mpm\" (UniqueName: \"kubernetes.io/projected/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-kube-api-access-c6mpm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.157119 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.156891    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.157119 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.156934    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.157119 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.156970    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.157338 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.157293    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.157338 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.157321    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.157427 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.157344    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.159345 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.159324    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.159614 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.159596    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.216937 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.216900    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6mpm\" (UniqueName: \"kubernetes.io/projected/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-kube-api-access-c6mpm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.303587 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.303553    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:36.727962 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.727885    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"]
Oct 31 00:49:36.733439 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:49:36.733411    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5365eaac_24ea_4fcf_9427_fddcdc7e0a74.slice/crio-66172e3b3aea228d98566d881319e2dd98eab096769ddb03aa6f059e8a13ca8b WatchSource:0}: Error finding container 66172e3b3aea228d98566d881319e2dd98eab096769ddb03aa6f059e8a13ca8b: Status 404 returned error can't find the container with id 66172e3b3aea228d98566d881319e2dd98eab096769ddb03aa6f059e8a13ca8b
Oct 31 00:49:36.735334 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.735318    4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:49:36.821400 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:36.821371    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh" event={"ID":"5365eaac-24ea-4fcf-9427-fddcdc7e0a74","Type":"ContainerStarted","Data":"66172e3b3aea228d98566d881319e2dd98eab096769ddb03aa6f059e8a13ca8b"}
Oct 31 00:49:39.838162 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:39.838124    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh" event={"ID":"5365eaac-24ea-4fcf-9427-fddcdc7e0a74","Type":"ContainerStarted","Data":"576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be"}
Oct 31 00:49:39.838628 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:39.838211    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:40.844152 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:40.844115    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh" event={"ID":"5365eaac-24ea-4fcf-9427-fddcdc7e0a74","Type":"ContainerStarted","Data":"8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c"}
Oct 31 00:49:40.959992 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:40.959937    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh" podStartSLOduration=3.471967781 podStartE2EDuration="5.959923046s" totalImagesPullingTime="2.487955265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:40.95962051 +0000 UTC m=+1253.142149804" watchObservedRunningTime="2026-10-31 00:49:40.959923046 +0000 UTC m=+1253.142452380"
Oct 31 00:49:41.303811 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:41.303772    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:44.152391 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:49:44.152349    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:49:46.303782 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:46.303745    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:46.317806 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:46.317779    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:46.883388 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:46.883355    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:49:57.891995 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:49:57.891966    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:50:13.583429 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.583395    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-qjhkm-pqscr"]
Oct 31 00:50:13.588462 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.588444    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.642709 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.642678    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qjhkm-pqscr"]
Oct 31 00:50:13.703687 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.703652    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a1ba1344-2386-4b51-9b13-72e5709907b0-archives-path\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.703687 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.703687    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-trusted-ca-bundle\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.703873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.703705    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-service-ca-bundle\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.703873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.703852    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k7wn\" (UniqueName: \"kubernetes.io/projected/a1ba1344-2386-4b51-9b13-72e5709907b0-kube-api-access-9k7wn\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.804594 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.804562    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k7wn\" (UniqueName: \"kubernetes.io/projected/a1ba1344-2386-4b51-9b13-72e5709907b0-kube-api-access-9k7wn\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.804750 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.804665    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a1ba1344-2386-4b51-9b13-72e5709907b0-archives-path\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.804750 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.804691    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-trusted-ca-bundle\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.804750 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.804718    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-service-ca-bundle\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.805125 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.805106    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a1ba1344-2386-4b51-9b13-72e5709907b0-archives-path\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.805443 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.805422    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-service-ca-bundle\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.805489 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.805426    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-trusted-ca-bundle\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.864734 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.864655    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k7wn\" (UniqueName: \"kubernetes.io/projected/a1ba1344-2386-4b51-9b13-72e5709907b0-kube-api-access-9k7wn\") pod \"periodic-gathering-qjhkm-pqscr\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") " pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:13.901780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:13.901752    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:50:14.152633 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:50:14.152544    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:50:14.272599 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:14.272570    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qjhkm-pqscr"]
Oct 31 00:50:14.279691 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:50:14.279659    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ba1344_2386_4b51_9b13_72e5709907b0.slice/crio-de6ed2c323e1b5b7ef25b6829399871cadc8894ba4fce0c1516c838ac4f0f086 WatchSource:0}: Error finding container de6ed2c323e1b5b7ef25b6829399871cadc8894ba4fce0c1516c838ac4f0f086: Status 404 returned error can't find the container with id de6ed2c323e1b5b7ef25b6829399871cadc8894ba4fce0c1516c838ac4f0f086
Oct 31 00:50:15.003835 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:15.003799    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qjhkm-pqscr" event={"ID":"a1ba1344-2386-4b51-9b13-72e5709907b0","Type":"ContainerStarted","Data":"3bd4cf5ffeec4b291270f5f10c4d9f505e04776f0763bcc932f5185bd0e5a0d7"}
Oct 31 00:50:15.004256 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:15.003848    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qjhkm-pqscr" event={"ID":"a1ba1344-2386-4b51-9b13-72e5709907b0","Type":"ContainerStarted","Data":"de6ed2c323e1b5b7ef25b6829399871cadc8894ba4fce0c1516c838ac4f0f086"}
Oct 31 00:50:15.120245 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:15.120194    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-qjhkm-pqscr" podStartSLOduration=2.120179856 podStartE2EDuration="2.120179856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:15.118556337 +0000 UTC m=+1287.301085634" watchObservedRunningTime="2026-10-31 00:50:15.120179856 +0000 UTC m=+1287.302709150"
Oct 31 00:50:20.267479 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.267442    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"]
Oct 31 00:50:20.278003 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:50:20.277967    4935 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c" cmd=["/bin/sleep","15"]
Oct 31 00:50:20.278176 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.278023    4935 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:50:20.278176 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:50:20.278057    4935 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh" podUID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" containerName="main" containerID="cri-o://8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c"
Oct 31 00:50:20.278176 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.278098    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh" podUID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" containerName="main" containerID="cri-o://8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c" gracePeriod=60
Oct 31 00:50:20.489109 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.489075    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh" podUID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" containerName="llm-d-routing-sidecar" containerID="cri-o://576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be" gracePeriod=60
Oct 31 00:50:20.747453 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.747427    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:50:20.869544 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.869471    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tls-certs\" (UniqueName: \"kubernetes.io/secret/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tls-certs\") pod \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") "
Oct 31 00:50:20.869544 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.869525    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tmp-dir\") pod \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") "
Oct 31 00:50:20.869762 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.869559    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-kube-api-access-c6mpm\" (UniqueName: \"kubernetes.io/projected/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-kube-api-access-c6mpm\") pod \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") "
Oct 31 00:50:20.869762 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.869582    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-dshm\") pod \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") "
Oct 31 00:50:20.869762 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.869651    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-model-cache\") pod \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") "
Oct 31 00:50:20.869762 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.869677    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-home\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-home\") pod \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\" (UID: \"5365eaac-24ea-4fcf-9427-fddcdc7e0a74\") "
Oct 31 00:50:20.869973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.869858    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tmp-dir" pod "5365eaac-24ea-4fcf-9427-fddcdc7e0a74" (UID: "5365eaac-24ea-4fcf-9427-fddcdc7e0a74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:20.869973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.869948    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:20.870099 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.870079    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-model-cache" pod "5365eaac-24ea-4fcf-9427-fddcdc7e0a74" (UID: "5365eaac-24ea-4fcf-9427-fddcdc7e0a74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:20.870220 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.870200    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-home" pod "5365eaac-24ea-4fcf-9427-fddcdc7e0a74" (UID: "5365eaac-24ea-4fcf-9427-fddcdc7e0a74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:20.872170 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.872140    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-dshm" pod "5365eaac-24ea-4fcf-9427-fddcdc7e0a74" (UID: "5365eaac-24ea-4fcf-9427-fddcdc7e0a74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:20.872267 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.872209    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tls-certs" pod "5365eaac-24ea-4fcf-9427-fddcdc7e0a74" (UID: "5365eaac-24ea-4fcf-9427-fddcdc7e0a74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:50:20.872267 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.872241    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-kube-api-access-c6mpm" pod "5365eaac-24ea-4fcf-9427-fddcdc7e0a74" (UID: "5365eaac-24ea-4fcf-9427-fddcdc7e0a74"). InnerVolumeSpecName "kube-api-access-c6mpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:20.970927 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.970893    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:20.970927 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.970921    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:20.971137 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.970934    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:20.971137 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.970946    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6mpm\" (UniqueName: \"kubernetes.io/projected/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-kube-api-access-c6mpm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:20.971137 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:20.970959    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5365eaac-24ea-4fcf-9427-fddcdc7e0a74-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:21.033733 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.033699    4935 generic.go:356] "Generic (PLEG): container finished" podID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" containerID="8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c" exitCode=0
Oct 31 00:50:21.033733 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.033727    4935 generic.go:356] "Generic (PLEG): container finished" podID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" containerID="576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be" exitCode=0
Oct 31 00:50:21.033919 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.033759    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"
Oct 31 00:50:21.033919 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.033776    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh" event={"ID":"5365eaac-24ea-4fcf-9427-fddcdc7e0a74","Type":"ContainerDied","Data":"8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c"}
Oct 31 00:50:21.033919 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.033830    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh" event={"ID":"5365eaac-24ea-4fcf-9427-fddcdc7e0a74","Type":"ContainerDied","Data":"576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be"}
Oct 31 00:50:21.033919 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.033846    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh" event={"ID":"5365eaac-24ea-4fcf-9427-fddcdc7e0a74","Type":"ContainerDied","Data":"66172e3b3aea228d98566d881319e2dd98eab096769ddb03aa6f059e8a13ca8b"}
Oct 31 00:50:21.033919 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.033865    4935 scope.go:122] "RemoveContainer" containerID="8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c"
Oct 31 00:50:21.043115 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.043099    4935 scope.go:122] "RemoveContainer" containerID="576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be"
Oct 31 00:50:21.050936 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.050917    4935 scope.go:122] "RemoveContainer" containerID="8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c"
Oct 31 00:50:21.051170 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:50:21.051152    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c\": container with ID starting with 8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c not found: ID does not exist" containerID="8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c"
Oct 31 00:50:21.051245 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.051180    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c"} err="failed to get container status \"8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c\": rpc error: code = NotFound desc = could not find container \"8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c\": container with ID starting with 8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c not found: ID does not exist"
Oct 31 00:50:21.051245 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.051195    4935 scope.go:122] "RemoveContainer" containerID="576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be"
Oct 31 00:50:21.051494 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:50:21.051473    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be\": container with ID starting with 576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be not found: ID does not exist" containerID="576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be"
Oct 31 00:50:21.051548 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.051502    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be"} err="failed to get container status \"576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be\": rpc error: code = NotFound desc = could not find container \"576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be\": container with ID starting with 576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be not found: ID does not exist"
Oct 31 00:50:21.051548 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.051516    4935 scope.go:122] "RemoveContainer" containerID="8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c"
Oct 31 00:50:21.051737 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.051720    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c"} err="failed to get container status \"8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c\": rpc error: code = NotFound desc = could not find container \"8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c\": container with ID starting with 8640137253ab3557fd47d1d37035200833a29bac58005b1ee06b6700d618fb5c not found: ID does not exist"
Oct 31 00:50:21.051806 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.051736    4935 scope.go:122] "RemoveContainer" containerID="576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be"
Oct 31 00:50:21.051995 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.051969    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be"} err="failed to get container status \"576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be\": rpc error: code = NotFound desc = could not find container \"576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be\": container with ID starting with 576e15624e2ce46337bd6ef8f33d018d59a4c6facb116d40e0e059f8d321b0be not found: ID does not exist"
Oct 31 00:50:21.193263 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.193186    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"]
Oct 31 00:50:21.198936 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.198911    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-v67nh"]
Oct 31 00:50:21.942886 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:50:21.942858    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" path="/var/lib/kubelet/pods/5365eaac-24ea-4fcf-9427-fddcdc7e0a74/volumes"
Oct 31 00:50:44.152558 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:50:44.152514    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:51:14.153021 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:51:14.152976    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:51:18.968887 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:18.968851    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"]
Oct 31 00:51:18.969298 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:18.969236    4935 cpu_manager.go:414] "containerMap: removing container" podUID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" containerName="main"
Oct 31 00:51:18.969298 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:18.969248    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" containerName="main"
Oct 31 00:51:18.969298 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:18.969266    4935 cpu_manager.go:414] "containerMap: removing container" podUID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" containerName="llm-d-routing-sidecar"
Oct 31 00:51:18.969298 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:18.969271    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" containerName="llm-d-routing-sidecar"
Oct 31 00:51:18.969423 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:18.969343    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" containerName="llm-d-routing-sidecar"
Oct 31 00:51:18.969423 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:18.969356    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="5365eaac-24ea-4fcf-9427-fddcdc7e0a74" containerName="main"
Oct 31 00:51:18.972589 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:18.972572    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.025500 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.025466    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.025642 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.025525    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.025642 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.025584    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8549\" (UniqueName: \"kubernetes.io/projected/18cac401-db6b-448a-b638-c0b2340d1fc4-kube-api-access-c8549\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.025715 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.025638    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.025715 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.025678    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18cac401-db6b-448a-b638-c0b2340d1fc4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.025776 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.025759    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.070238 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.070090    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Oct 31 00:51:19.070238 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.070108    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Oct 31 00:51:19.070238 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.070177    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-dgvwn\""
Oct 31 00:51:19.070238 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.070205    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Oct 31 00:51:19.070238 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.070233    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-grlwl\""
Oct 31 00:51:19.073025 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.073000    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"]
Oct 31 00:51:19.126919 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.126891    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.127124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.126948    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.127124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.126977    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8549\" (UniqueName: \"kubernetes.io/projected/18cac401-db6b-448a-b638-c0b2340d1fc4-kube-api-access-c8549\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.127124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.127006    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.127124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.127068    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18cac401-db6b-448a-b638-c0b2340d1fc4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.127124 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.127117    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.127389 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.127363    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.127461 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.127434    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-home\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.127519 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.127457    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.129448 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.129425    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.129779 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.129760    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18cac401-db6b-448a-b638-c0b2340d1fc4-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.187619 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.187587    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8549\" (UniqueName: \"kubernetes.io/projected/18cac401-db6b-448a-b638-c0b2340d1fc4-kube-api-access-c8549\") pod \"scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.285874 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.285839    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:19.711006 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:19.710919    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"]
Oct 31 00:51:19.717278 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:51:19.717252    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cac401_db6b_448a_b638_c0b2340d1fc4.slice/crio-24ebc125bf463bb3e6d7f403ab4cfc4b10a29afcdfa1c373dfcfcbe4ac5af632 WatchSource:0}: Error finding container 24ebc125bf463bb3e6d7f403ab4cfc4b10a29afcdfa1c373dfcfcbe4ac5af632: Status 404 returned error can't find the container with id 24ebc125bf463bb3e6d7f403ab4cfc4b10a29afcdfa1c373dfcfcbe4ac5af632
Oct 31 00:51:20.321323 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:20.321287    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw" event={"ID":"18cac401-db6b-448a-b638-c0b2340d1fc4","Type":"ContainerStarted","Data":"fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57"}
Oct 31 00:51:20.321814 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:20.321347    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw" event={"ID":"18cac401-db6b-448a-b638-c0b2340d1fc4","Type":"ContainerStarted","Data":"24ebc125bf463bb3e6d7f403ab4cfc4b10a29afcdfa1c373dfcfcbe4ac5af632"}
Oct 31 00:51:20.321814 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:20.321409    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:21.328089 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:21.328022    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw" event={"ID":"18cac401-db6b-448a-b638-c0b2340d1fc4","Type":"ContainerStarted","Data":"fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6"}
Oct 31 00:51:21.446288 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:21.446235    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw" podStartSLOduration=3.446222597 podStartE2EDuration="3.446222597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:21.444955139 +0000 UTC m=+1353.627484433" watchObservedRunningTime="2026-10-31 00:51:21.446222597 +0000 UTC m=+1353.628751922"
Oct 31 00:51:22.286008 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:22.285971    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:29.286734 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:29.286695    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:29.299552 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:29.299523    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:29.299709 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:29.299623    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:29.377159 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:29.377131    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:51:44.152207 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:51:44.152169    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:51:48.239016 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.238979    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"]
Oct 31 00:51:48.242987 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.242962    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.298449 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.298413    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Oct 31 00:51:48.300382 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.299980    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Oct 31 00:51:48.300876 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.300065    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Oct 31 00:51:48.300876 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.300535    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-envoy\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.301103 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.300987    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-credential-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.301103 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.301022    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2fa66c-25c5-4017-9d00-65bc10797cce-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.301103 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.301066    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-data\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.301228 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.301200    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-certs\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.301312 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.301296    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-podinfo\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.301375 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.301326    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbc7k\" (UniqueName: \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-kube-api-access-rbc7k\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.301426 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.301383    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.301474 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.301454    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-token\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.346844 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.346820    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-vcx7z\""
Oct 31 00:51:48.347019 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.346998    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"]
Oct 31 00:51:48.402096 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.402060    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-credential-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.402299 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.402104    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2fa66c-25c5-4017-9d00-65bc10797cce-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.402299 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.402250    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-data\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.402446 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.402290    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-certs\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.402446 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.402344    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-podinfo\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.402446 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.402374    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbc7k\" (UniqueName: \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-kube-api-access-rbc7k\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.402733 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.402697    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-certs\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.402823 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.402753    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.402823 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.402809    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-token\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.402932 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.402856    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-envoy\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.404076 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.404016    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2fa66c-25c5-4017-9d00-65bc10797cce-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.404524 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.404420    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-credential-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.404714 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.404694    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-data\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.404870 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.404815    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-socket\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.406508 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.406486    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-envoy\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.406939 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.406921    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-podinfo\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.464058 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.464000    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbc7k\" (UniqueName: \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-kube-api-access-rbc7k\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.464226 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.464110    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-token\") pod \"router-gateway-1-openshift-default-745b597ff7-7b52v\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.465737 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.465712    4935 generic.go:356] "Generic (PLEG): container finished" podID="a1ba1344-2386-4b51-9b13-72e5709907b0" containerID="3bd4cf5ffeec4b291270f5f10c4d9f505e04776f0763bcc932f5185bd0e5a0d7" exitCode=0
Oct 31 00:51:48.465862 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.465746    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qjhkm-pqscr" event={"ID":"a1ba1344-2386-4b51-9b13-72e5709907b0","Type":"ContainerDied","Data":"3bd4cf5ffeec4b291270f5f10c4d9f505e04776f0763bcc932f5185bd0e5a0d7"}
Oct 31 00:51:48.558000 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.557925    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:48.985654 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.985573    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"]
Oct 31 00:51:48.991745 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:51:48.991715    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2fa66c_25c5_4017_9d00_65bc10797cce.slice/crio-3d5daad52f9d53825f34a5e3e7404aff3b8ae7ff2f00cdc492edaf90ecc7ec73 WatchSource:0}: Error finding container 3d5daad52f9d53825f34a5e3e7404aff3b8ae7ff2f00cdc492edaf90ecc7ec73: Status 404 returned error can't find the container with id 3d5daad52f9d53825f34a5e3e7404aff3b8ae7ff2f00cdc492edaf90ecc7ec73
Oct 31 00:51:48.994426 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.994384    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:51:48.994535 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.994459    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:51:48.994535 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:48.994497    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 00:51:49.471799 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.471761    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v" event={"ID":"cc2fa66c-25c5-4017-9d00-65bc10797cce","Type":"ContainerStarted","Data":"14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d"}
Oct 31 00:51:49.472243 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.471819    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v" event={"ID":"cc2fa66c-25c5-4017-9d00-65bc10797cce","Type":"ContainerStarted","Data":"3d5daad52f9d53825f34a5e3e7404aff3b8ae7ff2f00cdc492edaf90ecc7ec73"}
Oct 31 00:51:49.558140 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.558109    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:49.588303 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.588253    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v" podStartSLOduration=1.588236301 podStartE2EDuration="1.588236301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:49.58812835 +0000 UTC m=+1381.770657643" watchObservedRunningTime="2026-10-31 00:51:49.588236301 +0000 UTC m=+1381.770765591"
Oct 31 00:51:49.611003 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.610976    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:51:49.716120 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.716086    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-trusted-ca-bundle\") pod \"a1ba1344-2386-4b51-9b13-72e5709907b0\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") "
Oct 31 00:51:49.716301 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.716142    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ba1344-2386-4b51-9b13-72e5709907b0-kube-api-access-9k7wn\" (UniqueName: \"kubernetes.io/projected/a1ba1344-2386-4b51-9b13-72e5709907b0-kube-api-access-9k7wn\") pod \"a1ba1344-2386-4b51-9b13-72e5709907b0\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") "
Oct 31 00:51:49.716301 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.716180    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ba1344-2386-4b51-9b13-72e5709907b0-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a1ba1344-2386-4b51-9b13-72e5709907b0-archives-path\") pod \"a1ba1344-2386-4b51-9b13-72e5709907b0\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") "
Oct 31 00:51:49.716301 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.716225    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-service-ca-bundle\") pod \"a1ba1344-2386-4b51-9b13-72e5709907b0\" (UID: \"a1ba1344-2386-4b51-9b13-72e5709907b0\") "
Oct 31 00:51:49.716508 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.716479    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ba1344-2386-4b51-9b13-72e5709907b0-archives-path" pod "a1ba1344-2386-4b51-9b13-72e5709907b0" (UID: "a1ba1344-2386-4b51-9b13-72e5709907b0"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:49.716633 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.716514    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-trusted-ca-bundle" pod "a1ba1344-2386-4b51-9b13-72e5709907b0" (UID: "a1ba1344-2386-4b51-9b13-72e5709907b0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:51:49.716633 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.716592    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-service-ca-bundle" pod "a1ba1344-2386-4b51-9b13-72e5709907b0" (UID: "a1ba1344-2386-4b51-9b13-72e5709907b0"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:51:49.718424 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.718406    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ba1344-2386-4b51-9b13-72e5709907b0-kube-api-access-9k7wn" pod "a1ba1344-2386-4b51-9b13-72e5709907b0" (UID: "a1ba1344-2386-4b51-9b13-72e5709907b0"). InnerVolumeSpecName "kube-api-access-9k7wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:49.816973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.816941    4935 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-trusted-ca-bundle\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:49.816973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.816968    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k7wn\" (UniqueName: \"kubernetes.io/projected/a1ba1344-2386-4b51-9b13-72e5709907b0-kube-api-access-9k7wn\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:49.816973 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.816980    4935 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a1ba1344-2386-4b51-9b13-72e5709907b0-archives-path\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:49.817257 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:49.816990    4935 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1ba1344-2386-4b51-9b13-72e5709907b0-service-ca-bundle\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:50.477522 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:50.477492    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qjhkm-pqscr"
Oct 31 00:51:50.478010 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:50.477530    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qjhkm-pqscr" event={"ID":"a1ba1344-2386-4b51-9b13-72e5709907b0","Type":"ContainerDied","Data":"de6ed2c323e1b5b7ef25b6829399871cadc8894ba4fce0c1516c838ac4f0f086"}
Oct 31 00:51:50.478010 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:50.477574    4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de6ed2c323e1b5b7ef25b6829399871cadc8894ba4fce0c1516c838ac4f0f086"
Oct 31 00:51:50.563529 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:50.563504    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:51.483525 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:51.483481    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:51:51.484344 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:51:51.484321    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:52:02.883523 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:02.883489    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"]
Oct 31 00:52:02.893558 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:52:02.893511    4935 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6" cmd=["/bin/sleep","15"]
Oct 31 00:52:02.893706 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:02.893585    4935 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:52:02.893706 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:52:02.893603    4935 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw" podUID="18cac401-db6b-448a-b638-c0b2340d1fc4" containerName="main" containerID="cri-o://fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6"
Oct 31 00:52:02.893706 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:02.893665    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw" podUID="18cac401-db6b-448a-b638-c0b2340d1fc4" containerName="main" containerID="cri-o://fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6" gracePeriod=60
Oct 31 00:52:03.111218 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.111181    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw" podUID="18cac401-db6b-448a-b638-c0b2340d1fc4" containerName="llm-d-routing-sidecar" containerID="cri-o://fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57" gracePeriod=60
Oct 31 00:52:03.367665 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.367638    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:52:03.444695 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.444667    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-model-cache\") pod \"18cac401-db6b-448a-b638-c0b2340d1fc4\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") "
Oct 31 00:52:03.444847 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.444701    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cac401-db6b-448a-b638-c0b2340d1fc4-tls-certs\" (UniqueName: \"kubernetes.io/secret/18cac401-db6b-448a-b638-c0b2340d1fc4-tls-certs\") pod \"18cac401-db6b-448a-b638-c0b2340d1fc4\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") "
Oct 31 00:52:03.444847 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.444740    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-dshm\") pod \"18cac401-db6b-448a-b638-c0b2340d1fc4\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") "
Oct 31 00:52:03.444847 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.444796    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-tmp-dir\") pod \"18cac401-db6b-448a-b638-c0b2340d1fc4\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") "
Oct 31 00:52:03.445013 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.444865    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cac401-db6b-448a-b638-c0b2340d1fc4-kube-api-access-c8549\" (UniqueName: \"kubernetes.io/projected/18cac401-db6b-448a-b638-c0b2340d1fc4-kube-api-access-c8549\") pod \"18cac401-db6b-448a-b638-c0b2340d1fc4\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") "
Oct 31 00:52:03.445013 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.444898    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-model-cache" pod "18cac401-db6b-448a-b638-c0b2340d1fc4" (UID: "18cac401-db6b-448a-b638-c0b2340d1fc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:03.445013 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.444903    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-home\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-home\") pod \"18cac401-db6b-448a-b638-c0b2340d1fc4\" (UID: \"18cac401-db6b-448a-b638-c0b2340d1fc4\") "
Oct 31 00:52:03.445184 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.445154    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-home" pod "18cac401-db6b-448a-b638-c0b2340d1fc4" (UID: "18cac401-db6b-448a-b638-c0b2340d1fc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:03.445184 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.445168    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-tmp-dir" pod "18cac401-db6b-448a-b638-c0b2340d1fc4" (UID: "18cac401-db6b-448a-b638-c0b2340d1fc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:03.445332 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.445314    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:03.445381 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.445337    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:03.445381 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.445350    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:03.447028 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.446998    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-dshm" pod "18cac401-db6b-448a-b638-c0b2340d1fc4" (UID: "18cac401-db6b-448a-b638-c0b2340d1fc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:03.447153 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.447052    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cac401-db6b-448a-b638-c0b2340d1fc4-tls-certs" pod "18cac401-db6b-448a-b638-c0b2340d1fc4" (UID: "18cac401-db6b-448a-b638-c0b2340d1fc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:52:03.447153 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.447113    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cac401-db6b-448a-b638-c0b2340d1fc4-kube-api-access-c8549" pod "18cac401-db6b-448a-b638-c0b2340d1fc4" (UID: "18cac401-db6b-448a-b638-c0b2340d1fc4"). InnerVolumeSpecName "kube-api-access-c8549". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:03.540190 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.540159    4935 generic.go:356] "Generic (PLEG): container finished" podID="18cac401-db6b-448a-b638-c0b2340d1fc4" containerID="fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6" exitCode=0
Oct 31 00:52:03.540190 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.540187    4935 generic.go:356] "Generic (PLEG): container finished" podID="18cac401-db6b-448a-b638-c0b2340d1fc4" containerID="fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57" exitCode=0
Oct 31 00:52:03.540393 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.540203    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"
Oct 31 00:52:03.540393 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.540245    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw" event={"ID":"18cac401-db6b-448a-b638-c0b2340d1fc4","Type":"ContainerDied","Data":"fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6"}
Oct 31 00:52:03.540393 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.540301    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw" event={"ID":"18cac401-db6b-448a-b638-c0b2340d1fc4","Type":"ContainerDied","Data":"fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57"}
Oct 31 00:52:03.540393 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.540317    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw" event={"ID":"18cac401-db6b-448a-b638-c0b2340d1fc4","Type":"ContainerDied","Data":"24ebc125bf463bb3e6d7f403ab4cfc4b10a29afcdfa1c373dfcfcbe4ac5af632"}
Oct 31 00:52:03.540393 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.540341    4935 scope.go:122] "RemoveContainer" containerID="fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6"
Oct 31 00:52:03.545803 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.545786    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18cac401-db6b-448a-b638-c0b2340d1fc4-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:03.545900 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.545805    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8549\" (UniqueName: \"kubernetes.io/projected/18cac401-db6b-448a-b638-c0b2340d1fc4-kube-api-access-c8549\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:03.545900 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.545815    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18cac401-db6b-448a-b638-c0b2340d1fc4-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:03.550162 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.550145    4935 scope.go:122] "RemoveContainer" containerID="fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57"
Oct 31 00:52:03.557946 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.557930    4935 scope.go:122] "RemoveContainer" containerID="fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6"
Oct 31 00:52:03.558248 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:52:03.558229    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6\": container with ID starting with fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6 not found: ID does not exist" containerID="fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6"
Oct 31 00:52:03.558331 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.558260    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6"} err="failed to get container status \"fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6\": rpc error: code = NotFound desc = could not find container \"fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6\": container with ID starting with fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6 not found: ID does not exist"
Oct 31 00:52:03.558331 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.558285    4935 scope.go:122] "RemoveContainer" containerID="fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57"
Oct 31 00:52:03.558559 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:52:03.558541    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57\": container with ID starting with fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57 not found: ID does not exist" containerID="fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57"
Oct 31 00:52:03.558597 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.558567    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57"} err="failed to get container status \"fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57\": rpc error: code = NotFound desc = could not find container \"fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57\": container with ID starting with fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57 not found: ID does not exist"
Oct 31 00:52:03.558597 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.558581    4935 scope.go:122] "RemoveContainer" containerID="fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6"
Oct 31 00:52:03.558808 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.558785    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6"} err="failed to get container status \"fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6\": rpc error: code = NotFound desc = could not find container \"fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6\": container with ID starting with fe34832c2e83d5304106af3d4897164a9d896bca675b0c8a90d1942c30f0aed6 not found: ID does not exist"
Oct 31 00:52:03.558881 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.558809    4935 scope.go:122] "RemoveContainer" containerID="fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57"
Oct 31 00:52:03.559018 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.559000    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57"} err="failed to get container status \"fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57\": rpc error: code = NotFound desc = could not find container \"fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57\": container with ID starting with fa441e7f0d416411dae122a476abae628906e63ec872c1fd2c17067c971cbe57 not found: ID does not exist"
Oct 31 00:52:03.699957 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.699924    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"]
Oct 31 00:52:03.705612 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.705589    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-bc5ffbf5b-prpkw"]
Oct 31 00:52:03.943295 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:03.943211    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18cac401-db6b-448a-b638-c0b2340d1fc4" path="/var/lib/kubelet/pods/18cac401-db6b-448a-b638-c0b2340d1fc4/volumes"
Oct 31 00:52:04.267314 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:04.267272    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"]
Oct 31 00:52:04.267524 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:04.267499    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v" podUID="cc2fa66c-25c5-4017-9d00-65bc10797cce" containerName="istio-proxy" containerID="cri-o://14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d" gracePeriod=30
Oct 31 00:52:05.392345 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.392311    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"]
Oct 31 00:52:05.392901 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.392884    4935 cpu_manager.go:414] "containerMap: removing container" podUID="18cac401-db6b-448a-b638-c0b2340d1fc4" containerName="llm-d-routing-sidecar"
Oct 31 00:52:05.392946 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.392904    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cac401-db6b-448a-b638-c0b2340d1fc4" containerName="llm-d-routing-sidecar"
Oct 31 00:52:05.392986 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.392944    4935 cpu_manager.go:414] "containerMap: removing container" podUID="a1ba1344-2386-4b51-9b13-72e5709907b0" containerName="insights-gathering"
Oct 31 00:52:05.392986 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.392952    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ba1344-2386-4b51-9b13-72e5709907b0" containerName="insights-gathering"
Oct 31 00:52:05.392986 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.392965    4935 cpu_manager.go:414] "containerMap: removing container" podUID="18cac401-db6b-448a-b638-c0b2340d1fc4" containerName="main"
Oct 31 00:52:05.392986 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.392973    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cac401-db6b-448a-b638-c0b2340d1fc4" containerName="main"
Oct 31 00:52:05.393116 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.393085    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cac401-db6b-448a-b638-c0b2340d1fc4" containerName="llm-d-routing-sidecar"
Oct 31 00:52:05.393116 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.393101    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ba1344-2386-4b51-9b13-72e5709907b0" containerName="insights-gathering"
Oct 31 00:52:05.393116 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.393109    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cac401-db6b-448a-b638-c0b2340d1fc4" containerName="main"
Oct 31 00:52:05.396782 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.396762    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.447176 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.447145    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"]
Oct 31 00:52:05.467206 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.467179    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tls-certs\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.467364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.467214    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1d7685f-c2ee-421a-9f4b-7340f478ed20-cabundle-cert\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.467364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.467312    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-home\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.467450 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.467361    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kserve-provision-location\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.467450 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.467393    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2th2p\" (UniqueName: \"kubernetes.io/projected/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kube-api-access-2th2p\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.467511 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.467463    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tmp-dir\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.467511 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.467488    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-dshm\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.467573 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.467542    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-model-cache\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.491883 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.491859    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Oct 31 00:52:05.492056 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.491889    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Oct 31 00:52:05.492056 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.491905    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Oct 31 00:52:05.492056 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.491921    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 00:52:05.492056 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.491955    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Oct 31 00:52:05.492056 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.491966    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-9cd6c\""
Oct 31 00:52:05.568517 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.568484    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tls-certs\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.568965 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.568528    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1d7685f-c2ee-421a-9f4b-7340f478ed20-cabundle-cert\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.568965 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.568562    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-home\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.568965 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.568601    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kserve-provision-location\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.568965 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.568642    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2th2p\" (UniqueName: \"kubernetes.io/projected/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kube-api-access-2th2p\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.568965 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.568671    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tmp-dir\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.568965 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.568694    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-dshm\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.568965 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.568795    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-model-cache\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.569418 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.569382    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tmp-dir\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.569463 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.569414    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-home\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.569506 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.569469    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kserve-provision-location\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.569542 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.569530    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-model-cache\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.569962 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.569931    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1d7685f-c2ee-421a-9f4b-7340f478ed20-cabundle-cert\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.571322 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.571302    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tls-certs\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.571849 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.571832    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-dshm\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.628885 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.628835    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2th2p\" (UniqueName: \"kubernetes.io/projected/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kube-api-access-2th2p\") pod \"auth-enabled-test-kserve-dd856666c-dpwls\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:05.715586 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:05.715508    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:06.144902 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:06.144857    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"]
Oct 31 00:52:06.150416 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:52:06.150389    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d7685f_c2ee_421a_9f4b_7340f478ed20.slice/crio-a0239d31d295506c7e5691b35c3dc4d481f1184a1e1b349f7f2593502c9c56d2 WatchSource:0}: Error finding container a0239d31d295506c7e5691b35c3dc4d481f1184a1e1b349f7f2593502c9c56d2: Status 404 returned error can't find the container with id a0239d31d295506c7e5691b35c3dc4d481f1184a1e1b349f7f2593502c9c56d2
Oct 31 00:52:06.486525 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:06.486483    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v" podUID="cc2fa66c-25c5-4017-9d00-65bc10797cce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:52:06.557143 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:06.557112    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" event={"ID":"e1d7685f-c2ee-421a-9f4b-7340f478ed20","Type":"ContainerStarted","Data":"56083a7bc4a0622a0df864dba6a52e7728d13e0e84bde83d96d5a99d09d8951c"}
Oct 31 00:52:06.557295 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:06.557162    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" event={"ID":"e1d7685f-c2ee-421a-9f4b-7340f478ed20","Type":"ContainerStarted","Data":"a0239d31d295506c7e5691b35c3dc4d481f1184a1e1b349f7f2593502c9c56d2"}
Oct 31 00:52:08.570292 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:08.570206    4935 generic.go:356] "Generic (PLEG): container finished" podID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerID="56083a7bc4a0622a0df864dba6a52e7728d13e0e84bde83d96d5a99d09d8951c" exitCode=0
Oct 31 00:52:08.570292 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:08.570277    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" event={"ID":"e1d7685f-c2ee-421a-9f4b-7340f478ed20","Type":"ContainerDied","Data":"56083a7bc4a0622a0df864dba6a52e7728d13e0e84bde83d96d5a99d09d8951c"}
Oct 31 00:52:09.528954 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.528897    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:52:09.575884 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.575849    4935 generic.go:356] "Generic (PLEG): container finished" podID="cc2fa66c-25c5-4017-9d00-65bc10797cce" containerID="14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d" exitCode=0
Oct 31 00:52:09.576272 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.575893    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v" event={"ID":"cc2fa66c-25c5-4017-9d00-65bc10797cce","Type":"ContainerDied","Data":"14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d"}
Oct 31 00:52:09.576272 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.575897    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"
Oct 31 00:52:09.576272 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.575932    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v" event={"ID":"cc2fa66c-25c5-4017-9d00-65bc10797cce","Type":"ContainerDied","Data":"3d5daad52f9d53825f34a5e3e7404aff3b8ae7ff2f00cdc492edaf90ecc7ec73"}
Oct 31 00:52:09.576272 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.575953    4935 scope.go:122] "RemoveContainer" containerID="14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d"
Oct 31 00:52:09.584531 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.584513    4935 scope.go:122] "RemoveContainer" containerID="14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d"
Oct 31 00:52:09.584768 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:52:09.584753    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d\": container with ID starting with 14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d not found: ID does not exist" containerID="14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d"
Oct 31 00:52:09.584815 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.584777    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d"} err="failed to get container status \"14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d\": rpc error: code = NotFound desc = could not find container \"14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d\": container with ID starting with 14c4bc85a4ecd73311f53108db2faed5ed44af7a5eb247159fd667c4219ccf1d not found: ID does not exist"
Oct 31 00:52:09.616232 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616211    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-data\") pod \"cc2fa66c-25c5-4017-9d00-65bc10797cce\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") "
Oct 31 00:52:09.616354 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616255    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-kube-api-access-rbc7k\" (UniqueName: \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-kube-api-access-rbc7k\") pod \"cc2fa66c-25c5-4017-9d00-65bc10797cce\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") "
Oct 31 00:52:09.616354 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616272    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-socket\") pod \"cc2fa66c-25c5-4017-9d00-65bc10797cce\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") "
Oct 31 00:52:09.616354 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616302    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-certs\") pod \"cc2fa66c-25c5-4017-9d00-65bc10797cce\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") "
Oct 31 00:52:09.616503 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616367    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-token\" (UniqueName: \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-token\") pod \"cc2fa66c-25c5-4017-9d00-65bc10797cce\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") "
Oct 31 00:52:09.616503 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616398    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-podinfo\") pod \"cc2fa66c-25c5-4017-9d00-65bc10797cce\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") "
Oct 31 00:52:09.616503 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616441    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-envoy\") pod \"cc2fa66c-25c5-4017-9d00-65bc10797cce\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") "
Oct 31 00:52:09.616503 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616480    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-credential-socket\") pod \"cc2fa66c-25c5-4017-9d00-65bc10797cce\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") "
Oct 31 00:52:09.616695 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616525    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2fa66c-25c5-4017-9d00-65bc10797cce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2fa66c-25c5-4017-9d00-65bc10797cce-istiod-ca-cert\") pod \"cc2fa66c-25c5-4017-9d00-65bc10797cce\" (UID: \"cc2fa66c-25c5-4017-9d00-65bc10797cce\") "
Oct 31 00:52:09.616748 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616520    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-data" pod "cc2fa66c-25c5-4017-9d00-65bc10797cce" (UID: "cc2fa66c-25c5-4017-9d00-65bc10797cce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:09.616748 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616547    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-socket" pod "cc2fa66c-25c5-4017-9d00-65bc10797cce" (UID: "cc2fa66c-25c5-4017-9d00-65bc10797cce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:09.616849 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616661    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-certs" pod "cc2fa66c-25c5-4017-9d00-65bc10797cce" (UID: "cc2fa66c-25c5-4017-9d00-65bc10797cce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:09.616849 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616782    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-credential-socket" pod "cc2fa66c-25c5-4017-9d00-65bc10797cce" (UID: "cc2fa66c-25c5-4017-9d00-65bc10797cce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:09.616849 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616823    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:09.616849 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616839    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:09.616849 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.616848    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:09.617297 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.617235    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2fa66c-25c5-4017-9d00-65bc10797cce-istiod-ca-cert" pod "cc2fa66c-25c5-4017-9d00-65bc10797cce" (UID: "cc2fa66c-25c5-4017-9d00-65bc10797cce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:52:09.618664 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.618641    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-token" pod "cc2fa66c-25c5-4017-9d00-65bc10797cce" (UID: "cc2fa66c-25c5-4017-9d00-65bc10797cce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:09.618743 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.618666    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-kube-api-access-rbc7k" pod "cc2fa66c-25c5-4017-9d00-65bc10797cce" (UID: "cc2fa66c-25c5-4017-9d00-65bc10797cce"). InnerVolumeSpecName "kube-api-access-rbc7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:09.618743 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.618680    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-podinfo" pod "cc2fa66c-25c5-4017-9d00-65bc10797cce" (UID: "cc2fa66c-25c5-4017-9d00-65bc10797cce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:52:09.618823 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.618750    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-envoy" pod "cc2fa66c-25c5-4017-9d00-65bc10797cce" (UID: "cc2fa66c-25c5-4017-9d00-65bc10797cce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:09.718330 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.718302    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2fa66c-25c5-4017-9d00-65bc10797cce-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:09.718330 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.718330    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbc7k\" (UniqueName: \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-kube-api-access-rbc7k\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:09.718518 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.718343    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:09.718518 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.718355    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:09.718518 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.718367    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:09.718518 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.718377    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2fa66c-25c5-4017-9d00-65bc10797cce-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:09.994279 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:09.994242    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"]
Oct 31 00:52:10.001096 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:10.001068    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-745b597ff7-7b52v"]
Oct 31 00:52:11.946612 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:11.946217    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2fa66c-25c5-4017-9d00-65bc10797cce" path="/var/lib/kubelet/pods/cc2fa66c-25c5-4017-9d00-65bc10797cce/volumes"
Oct 31 00:52:14.158818 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:52:14.158777    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:52:37.742743 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:37.742708    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" event={"ID":"e1d7685f-c2ee-421a-9f4b-7340f478ed20","Type":"ContainerStarted","Data":"64b32714632a03699f6b5e85b5e74202a1ac981a3405bf7333eace80de8ca926"}
Oct 31 00:52:37.959594 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:37.959545    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podStartSLOduration=2.259437412 podStartE2EDuration="32.959532144s" totalImagesPullingTime="28.635699056s" totalInitContainerRuntime="2.064395676s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:37.907620303 +0000 UTC m=+1430.090149599" watchObservedRunningTime="2026-10-31 00:52:37.959532144 +0000 UTC m=+1430.142061437"
Oct 31 00:52:38.716006 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:38.715962    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:44.152584 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:52:44.152542    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:52:45.715931 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:45.715893    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:52:45.717284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:45.717256    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 31 00:52:55.716554 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:55.716511    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 31 00:52:58.303756 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.303714    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"]
Oct 31 00:52:58.304296 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.304120    4935 cpu_manager.go:414] "containerMap: removing container" podUID="cc2fa66c-25c5-4017-9d00-65bc10797cce" containerName="istio-proxy"
Oct 31 00:52:58.304296 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.304133    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2fa66c-25c5-4017-9d00-65bc10797cce" containerName="istio-proxy"
Oct 31 00:52:58.304296 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.304208    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2fa66c-25c5-4017-9d00-65bc10797cce" containerName="istio-proxy"
Oct 31 00:52:58.366263 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.366228    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.397587 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.397555    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"]
Oct 31 00:52:58.443635 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.443606    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Oct 31 00:52:58.443822 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.443625    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Oct 31 00:52:58.443822 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.443634    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Oct 31 00:52:58.443822 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.443673    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 00:52:58.463452 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.463418    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-dw7n5\""
Oct 31 00:52:58.463452 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.463417    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Oct 31 00:52:58.503396 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.503363    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.503568 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.503413    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.503568 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.503497    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.503568 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.503544    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.503670 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.503608    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.503670 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.503633    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-home\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.503733 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.503676    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gxnb\" (UniqueName: \"kubernetes.io/projected/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kube-api-access-2gxnb\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.503817 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.503798    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-dshm\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.604706 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.604615    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.604706 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.604652    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-home\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.604706 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.604689    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gxnb\" (UniqueName: \"kubernetes.io/projected/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kube-api-access-2gxnb\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.604994 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.604741    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-dshm\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.604994 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.604773    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.604994 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.604793    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.604994 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.604814    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.604994 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.604857    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.605272 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.605163    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-home\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.605330 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.605274    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.605330 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.605319    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.605418 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.605402    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.605794 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.605703    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.607326 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.607301    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-dshm\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.607546 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.607527    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.665003 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.664969    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gxnb\" (UniqueName: \"kubernetes.io/projected/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kube-api-access-2gxnb\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-b942j\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:58.681837 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:58.681805    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:52:59.107807 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.107778    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"]
Oct 31 00:52:59.114161 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:52:59.114137    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80c1860_fe38_4206_ae4b_d8d2ebf1717a.slice/crio-c3c82e3be075ff29bb03fd4a6c814cdecd371d2b8dc02066c0b901a00e09bcd2 WatchSource:0}: Error finding container c3c82e3be075ff29bb03fd4a6c814cdecd371d2b8dc02066c0b901a00e09bcd2: Status 404 returned error can't find the container with id c3c82e3be075ff29bb03fd4a6c814cdecd371d2b8dc02066c0b901a00e09bcd2
Oct 31 00:52:59.181505 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.181475    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"]
Oct 31 00:52:59.192149 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.192114    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.244060 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.244000    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"]
Oct 31 00:52:59.312402 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.312366    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.312815 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.312461    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.312815 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.312549    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.312815 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.312593    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.413381 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.413297    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.413381 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.413346    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.413600 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.413471    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.413600 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.413567    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.413801 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.413783    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.413872 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.413834    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.414185 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.414158    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.416392 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.416365    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-847989858c-2rf88\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.507817 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.507785    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:52:59.857675 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.857631    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j" event={"ID":"e80c1860-fe38-4206-ae4b-d8d2ebf1717a","Type":"ContainerStarted","Data":"d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379"}
Oct 31 00:52:59.857855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.857702    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j" event={"ID":"e80c1860-fe38-4206-ae4b-d8d2ebf1717a","Type":"ContainerStarted","Data":"c3c82e3be075ff29bb03fd4a6c814cdecd371d2b8dc02066c0b901a00e09bcd2"}
Oct 31 00:52:59.892060 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:52:59.892003    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"]
Oct 31 00:52:59.897566 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:52:59.897521    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fbaf37_1e28_48fa_b328_6e1b6b8f9bba.slice/crio-e929b558eb40b0ba9520f252c402aedcd5c3e4ead45bd2cde9eb15d70555103b WatchSource:0}: Error finding container e929b558eb40b0ba9520f252c402aedcd5c3e4ead45bd2cde9eb15d70555103b: Status 404 returned error can't find the container with id e929b558eb40b0ba9520f252c402aedcd5c3e4ead45bd2cde9eb15d70555103b
Oct 31 00:53:00.881181 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:00.881074    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88" event={"ID":"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba","Type":"ContainerStarted","Data":"00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5"}
Oct 31 00:53:00.881181 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:00.881140    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88" event={"ID":"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba","Type":"ContainerStarted","Data":"e929b558eb40b0ba9520f252c402aedcd5c3e4ead45bd2cde9eb15d70555103b"}
Oct 31 00:53:01.890965 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:01.890859    4935 generic.go:356] "Generic (PLEG): container finished" podID="e80c1860-fe38-4206-ae4b-d8d2ebf1717a" containerID="d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379" exitCode=0
Oct 31 00:53:01.891612 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:01.891576    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j" event={"ID":"e80c1860-fe38-4206-ae4b-d8d2ebf1717a","Type":"ContainerDied","Data":"d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379"}
Oct 31 00:53:02.906592 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:02.905287    4935 generic.go:356] "Generic (PLEG): container finished" podID="29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" containerID="00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5" exitCode=0
Oct 31 00:53:02.906592 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:02.905417    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88" event={"ID":"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba","Type":"ContainerDied","Data":"00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5"}
Oct 31 00:53:02.924136 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:02.924097    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j" event={"ID":"e80c1860-fe38-4206-ae4b-d8d2ebf1717a","Type":"ContainerStarted","Data":"b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350"}
Oct 31 00:53:03.140861 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:03.140794    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j" podStartSLOduration=2.6892376159999998 podStartE2EDuration="5.140774103s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.451536487s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:03.139079887 +0000 UTC m=+1455.321609173" watchObservedRunningTime="2026-10-31 00:53:03.140774103 +0000 UTC m=+1455.323303397"
Oct 31 00:53:03.682830 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:03.682790    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:53:03.932652 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:03.932612    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88" event={"ID":"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba","Type":"ContainerStarted","Data":"e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56"}
Oct 31 00:53:03.933149 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:03.932987    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:53:04.049715 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:04.049660    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88" podStartSLOduration=2.622918594 podStartE2EDuration="5.049645195s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.426726601s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:04.04791443 +0000 UTC m=+1456.230443723" watchObservedRunningTime="2026-10-31 00:53:04.049645195 +0000 UTC m=+1456.232174529"
Oct 31 00:53:05.716247 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:05.716200    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 31 00:53:08.682947 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:08.682906    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:53:08.695428 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:08.695399    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:53:08.978641 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:08.978561    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:53:14.154090 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:53:14.154024    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:53:15.716451 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:15.716406    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 31 00:53:25.716783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:25.716725    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 31 00:53:34.949534 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:34.949449    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:53:35.716627 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:35.716583    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 31 00:53:39.259080 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:39.259031    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-847989858c-2rf88_29fbaf37-1e28-48fa-b328-6e1b6b8f9bba/main/0.log"
Oct 31 00:53:43.254351 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.254309    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"]
Oct 31 00:53:43.261160 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.261129    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"]
Oct 31 00:53:43.261440 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.261385    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88" podUID="29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" containerName="main" containerID="cri-o://e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56" gracePeriod=30
Oct 31 00:53:43.265344 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:53:43.265303    4935 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350" cmd=["/bin/sleep","15"]
Oct 31 00:53:43.265491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.265370    4935 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:53:43.265491 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:53:43.265387    4935 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j" podUID="e80c1860-fe38-4206-ae4b-d8d2ebf1717a" containerName="main" containerID="cri-o://b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350"
Oct 31 00:53:43.265491 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.265431    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j" podUID="e80c1860-fe38-4206-ae4b-d8d2ebf1717a" containerName="main" containerID="cri-o://b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350" gracePeriod=60
Oct 31 00:53:43.644722 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.644696    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:53:43.749798 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.749771    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-model-cache\") pod \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") "
Oct 31 00:53:43.750165 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.750141    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tmp-dir\") pod \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") "
Oct 31 00:53:43.750264 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.750182    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-dshm\") pod \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") "
Oct 31 00:53:43.750264 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.750214    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-home\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-home\") pod \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") "
Oct 31 00:53:43.750264 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.750257    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kserve-provision-location\") pod \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") "
Oct 31 00:53:43.750401 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.750337    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-cabundle-cert\") pod \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") "
Oct 31 00:53:43.750401 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.750367    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tls-certs\") pod \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") "
Oct 31 00:53:43.750474 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.750412    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kube-api-access-2gxnb\" (UniqueName: \"kubernetes.io/projected/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kube-api-access-2gxnb\") pod \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\" (UID: \"e80c1860-fe38-4206-ae4b-d8d2ebf1717a\") "
Oct 31 00:53:43.750918 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.750893    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-home" pod "e80c1860-fe38-4206-ae4b-d8d2ebf1717a" (UID: "e80c1860-fe38-4206-ae4b-d8d2ebf1717a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:43.751117 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.751087    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-model-cache" pod "e80c1860-fe38-4206-ae4b-d8d2ebf1717a" (UID: "e80c1860-fe38-4206-ae4b-d8d2ebf1717a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:43.751237 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.751192    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tmp-dir" pod "e80c1860-fe38-4206-ae4b-d8d2ebf1717a" (UID: "e80c1860-fe38-4206-ae4b-d8d2ebf1717a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:43.751368 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.751345    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-cabundle-cert" pod "e80c1860-fe38-4206-ae4b-d8d2ebf1717a" (UID: "e80c1860-fe38-4206-ae4b-d8d2ebf1717a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:43.754436 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.754409    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tls-certs" pod "e80c1860-fe38-4206-ae4b-d8d2ebf1717a" (UID: "e80c1860-fe38-4206-ae4b-d8d2ebf1717a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:53:43.754436 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.754423    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-dshm" pod "e80c1860-fe38-4206-ae4b-d8d2ebf1717a" (UID: "e80c1860-fe38-4206-ae4b-d8d2ebf1717a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:43.757210 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.757186    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kube-api-access-2gxnb" pod "e80c1860-fe38-4206-ae4b-d8d2ebf1717a" (UID: "e80c1860-fe38-4206-ae4b-d8d2ebf1717a"). InnerVolumeSpecName "kube-api-access-2gxnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:43.783326 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.783287    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kserve-provision-location" pod "e80c1860-fe38-4206-ae4b-d8d2ebf1717a" (UID: "e80c1860-fe38-4206-ae4b-d8d2ebf1717a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:43.834756 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.834733    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:53:43.852129 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.852106    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.852129 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.852128    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.852296 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.852140    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.852296 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.852151    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.852296 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.852159    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gxnb\" (UniqueName: \"kubernetes.io/projected/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-kube-api-access-2gxnb\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.852296 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.852167    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.852296 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.852176    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.852296 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.852183    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80c1860-fe38-4206-ae4b-d8d2ebf1717a-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.952539 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.952513    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tls-certs\" (UniqueName: \"kubernetes.io/secret/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tls-certs\") pod \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") "
Oct 31 00:53:43.952717 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.952570    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tmp-dir\") pod \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") "
Oct 31 00:53:43.952717 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.952633    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-kserve-provision-location\") pod \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") "
Oct 31 00:53:43.952717 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.952664    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-cabundle-cert\") pod \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\" (UID: \"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba\") "
Oct 31 00:53:43.953515 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.953486    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-cabundle-cert" pod "29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" (UID: "29fbaf37-1e28-48fa-b328-6e1b6b8f9bba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:43.954089 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.954055    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tmp-dir" pod "29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" (UID: "29fbaf37-1e28-48fa-b328-6e1b6b8f9bba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:43.956299 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.956271    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tls-certs" pod "29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" (UID: "29fbaf37-1e28-48fa-b328-6e1b6b8f9bba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:53:43.974696 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:43.974663    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-kserve-provision-location" pod "29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" (UID: "29fbaf37-1e28-48fa-b328-6e1b6b8f9bba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:44.054118 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.054085    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:44.054118 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.054116    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:44.054321 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.054129    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:44.054321 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.054143    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:44.147626 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.147547    4935 generic.go:356] "Generic (PLEG): container finished" podID="29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" containerID="e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56" exitCode=0
Oct 31 00:53:44.147626 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.147600    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"
Oct 31 00:53:44.147854 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.147670    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88" event={"ID":"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba","Type":"ContainerDied","Data":"e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56"}
Oct 31 00:53:44.147854 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.147725    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88" event={"ID":"29fbaf37-1e28-48fa-b328-6e1b6b8f9bba","Type":"ContainerDied","Data":"e929b558eb40b0ba9520f252c402aedcd5c3e4ead45bd2cde9eb15d70555103b"}
Oct 31 00:53:44.147854 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.147753    4935 scope.go:122] "RemoveContainer" containerID="e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56"
Oct 31 00:53:44.149836 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.149797    4935 generic.go:356] "Generic (PLEG): container finished" podID="e80c1860-fe38-4206-ae4b-d8d2ebf1717a" containerID="b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350" exitCode=0
Oct 31 00:53:44.149953 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.149908    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"
Oct 31 00:53:44.149953 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.149917    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j" event={"ID":"e80c1860-fe38-4206-ae4b-d8d2ebf1717a","Type":"ContainerDied","Data":"b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350"}
Oct 31 00:53:44.150105 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.149961    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j" event={"ID":"e80c1860-fe38-4206-ae4b-d8d2ebf1717a","Type":"ContainerDied","Data":"c3c82e3be075ff29bb03fd4a6c814cdecd371d2b8dc02066c0b901a00e09bcd2"}
Oct 31 00:53:44.158414 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.158398    4935 scope.go:122] "RemoveContainer" containerID="00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5"
Oct 31 00:53:44.161623 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:53:44.161598    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:53:44.167788 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.167774    4935 scope.go:122] "RemoveContainer" containerID="e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56"
Oct 31 00:53:44.168047 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:53:44.168019    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56\": container with ID starting with e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56 not found: ID does not exist" containerID="e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56"
Oct 31 00:53:44.168103 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.168070    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56"} err="failed to get container status \"e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56\": rpc error: code = NotFound desc = could not find container \"e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56\": container with ID starting with e1568b839a8ad2b3b32147556e47ecc5613268b009aa0c87114e03a193f69d56 not found: ID does not exist"
Oct 31 00:53:44.168103 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.168092    4935 scope.go:122] "RemoveContainer" containerID="00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5"
Oct 31 00:53:44.168293 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:53:44.168279    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5\": container with ID starting with 00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5 not found: ID does not exist" containerID="00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5"
Oct 31 00:53:44.168337 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.168298    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5"} err="failed to get container status \"00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5\": rpc error: code = NotFound desc = could not find container \"00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5\": container with ID starting with 00e44a382b3786b2e4f109a00a902545fd0c51c7c9fd5d1c11c4d0d82c5081b5 not found: ID does not exist"
Oct 31 00:53:44.168337 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.168311    4935 scope.go:122] "RemoveContainer" containerID="b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350"
Oct 31 00:53:44.178554 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.178537    4935 scope.go:122] "RemoveContainer" containerID="d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379"
Oct 31 00:53:44.186626 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.186605    4935 scope.go:122] "RemoveContainer" containerID="b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350"
Oct 31 00:53:44.186933 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:53:44.186914    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350\": container with ID starting with b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350 not found: ID does not exist" containerID="b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350"
Oct 31 00:53:44.186995 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.186946    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350"} err="failed to get container status \"b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350\": rpc error: code = NotFound desc = could not find container \"b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350\": container with ID starting with b9cebab84f0eb0dc1b083bf5f21eca98ac63b52bed37164e8409a2488224a350 not found: ID does not exist"
Oct 31 00:53:44.186995 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.186969    4935 scope.go:122] "RemoveContainer" containerID="d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379"
Oct 31 00:53:44.187263 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:53:44.187246    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379\": container with ID starting with d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379 not found: ID does not exist" containerID="d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379"
Oct 31 00:53:44.187321 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.187271    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379"} err="failed to get container status \"d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379\": rpc error: code = NotFound desc = could not find container \"d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379\": container with ID starting with d8fb6aeec33308dfeb956ddc82dec4ea014c67d9b551e9ffe6838c478c555379 not found: ID does not exist"
Oct 31 00:53:44.310980 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.310939    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"]
Oct 31 00:53:44.317420 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.317389    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-847989858c-2rf88"]
Oct 31 00:53:44.424179 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.424104    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"]
Oct 31 00:53:44.429583 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:44.429557    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-b942j"]
Oct 31 00:53:45.716466 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:45.716417    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 31 00:53:45.948374 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:45.945592    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" path="/var/lib/kubelet/pods/29fbaf37-1e28-48fa-b328-6e1b6b8f9bba/volumes"
Oct 31 00:53:45.948374 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:45.946308    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80c1860-fe38-4206-ae4b-d8d2ebf1717a" path="/var/lib/kubelet/pods/e80c1860-fe38-4206-ae4b-d8d2ebf1717a/volumes"
Oct 31 00:53:48.080476 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:48.080449    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:53:48.088513 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:48.088493    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:53:55.715941 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:53:55.715898    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 31 00:54:05.716054 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:05.715992    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 31 00:54:14.154192 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:54:14.154154    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:54:15.716141 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:15.716092    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 31 00:54:25.725981 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:25.725951    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:54:25.733351 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:25.733327    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:54:35.831725 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:35.831685    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"]
Oct 31 00:54:36.197588 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.197511    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"]
Oct 31 00:54:36.197922 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.197910    4935 cpu_manager.go:414] "containerMap: removing container" podUID="29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" containerName="storage-initializer"
Oct 31 00:54:36.197967 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.197925    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" containerName="storage-initializer"
Oct 31 00:54:36.197967 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.197936    4935 cpu_manager.go:414] "containerMap: removing container" podUID="29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" containerName="main"
Oct 31 00:54:36.197967 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.197941    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" containerName="main"
Oct 31 00:54:36.197967 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.197953    4935 cpu_manager.go:414] "containerMap: removing container" podUID="e80c1860-fe38-4206-ae4b-d8d2ebf1717a" containerName="storage-initializer"
Oct 31 00:54:36.197967 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.197958    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80c1860-fe38-4206-ae4b-d8d2ebf1717a" containerName="storage-initializer"
Oct 31 00:54:36.198177 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.197971    4935 cpu_manager.go:414] "containerMap: removing container" podUID="e80c1860-fe38-4206-ae4b-d8d2ebf1717a" containerName="main"
Oct 31 00:54:36.198177 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.197976    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80c1860-fe38-4206-ae4b-d8d2ebf1717a" containerName="main"
Oct 31 00:54:36.198177 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.198030    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fbaf37-1e28-48fa-b328-6e1b6b8f9bba" containerName="main"
Oct 31 00:54:36.198177 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.198057    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80c1860-fe38-4206-ae4b-d8d2ebf1717a" containerName="main"
Oct 31 00:54:36.203716 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.203691    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.232657 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.232626    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-dshm\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.232787 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.232676    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-model-cache\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.232787 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.232749    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.232861 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.232786    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.232903 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.232861    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.232937 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.232915    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p5qw\" (UniqueName: \"kubernetes.io/projected/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kube-api-access-6p5qw\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.232969 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.232941    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.233021 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.232984    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-home\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.260015 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.259988    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"]
Oct 31 00:54:36.285704 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.285674    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Oct 31 00:54:36.285848 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.285829    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-5nnvq\""
Oct 31 00:54:36.285895 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.285829    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Oct 31 00:54:36.334226 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334197    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5qw\" (UniqueName: \"kubernetes.io/projected/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kube-api-access-6p5qw\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.334226 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334229    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.334486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334250    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-home\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.334486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334281    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-dshm\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.334486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334304    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-model-cache\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.334486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334350    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.334486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334387    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.334486 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334432    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.334733 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334680    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-home\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.334733 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334701    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-model-cache\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.334837 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334792    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.334837 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.334809    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.335677 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.335658    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 00:54:36.335807 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.335785    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Oct 31 00:54:36.335856 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.335825    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Oct 31 00:54:36.336695 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.336676    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-dshm\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.337141 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.337124    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.345423 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.345400    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.394132 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.394097    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5qw\" (UniqueName: \"kubernetes.io/projected/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kube-api-access-6p5qw\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-gqhwb\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.522481 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.522444    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:36.847730 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.847651    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"]
Oct 31 00:54:36.851431 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.851414    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:36.908506 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.908478    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"]
Oct 31 00:54:36.939984 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.939939    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:36.940186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.940017    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:36.940186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.940069    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:36.940186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.940165    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:36.947517 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:36.947493    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"]
Oct 31 00:54:36.953669 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:54:36.953641    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3726aa_12d4_4321_88d2_b2f77b6ff1fc.slice/crio-d6919a569433ebbd4b8972e04123f1b622e3eb9f75c834ded33b0258a758b41a WatchSource:0}: Error finding container d6919a569433ebbd4b8972e04123f1b622e3eb9f75c834ded33b0258a758b41a: Status 404 returned error can't find the container with id d6919a569433ebbd4b8972e04123f1b622e3eb9f75c834ded33b0258a758b41a
Oct 31 00:54:37.041359 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.041316    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:37.041515 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.041435    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:37.041515 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.041499    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:37.041613 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.041523    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:37.041890 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.041857    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:37.041995 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.041909    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:37.042399 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.042376    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:37.044324 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.044304    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:37.166236 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.166152    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:37.412500 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.412453    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb" event={"ID":"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc","Type":"ContainerStarted","Data":"bfec38cab9963d545e0bab0f06231a6e499e2790ca7007c315b6262229c0fe6c"}
Oct 31 00:54:37.412670 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.412522    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb" event={"ID":"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc","Type":"ContainerStarted","Data":"d6919a569433ebbd4b8972e04123f1b622e3eb9f75c834ded33b0258a758b41a"}
Oct 31 00:54:37.545405 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:54:37.545375    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853937a2_9ba9_4c14_a25d_90ecd6a8a08a.slice/crio-4ea7efe3b5e08df1b152284fb805b80e3e8fb28d21c9a6f118b3ca511d365d7e WatchSource:0}: Error finding container 4ea7efe3b5e08df1b152284fb805b80e3e8fb28d21c9a6f118b3ca511d365d7e: Status 404 returned error can't find the container with id 4ea7efe3b5e08df1b152284fb805b80e3e8fb28d21c9a6f118b3ca511d365d7e
Oct 31 00:54:37.579122 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:37.579091    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"]
Oct 31 00:54:38.419873 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:38.419790    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk" event={"ID":"853937a2-9ba9-4c14-a25d-90ecd6a8a08a","Type":"ContainerStarted","Data":"a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387"}
Oct 31 00:54:38.420375 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:38.419929    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk" event={"ID":"853937a2-9ba9-4c14-a25d-90ecd6a8a08a","Type":"ContainerStarted","Data":"4ea7efe3b5e08df1b152284fb805b80e3e8fb28d21c9a6f118b3ca511d365d7e"}
Oct 31 00:54:39.425930 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:39.425896    4935 generic.go:356] "Generic (PLEG): container finished" podID="1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" containerID="bfec38cab9963d545e0bab0f06231a6e499e2790ca7007c315b6262229c0fe6c" exitCode=0
Oct 31 00:54:39.426413 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:39.425977    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb" event={"ID":"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc","Type":"ContainerDied","Data":"bfec38cab9963d545e0bab0f06231a6e499e2790ca7007c315b6262229c0fe6c"}
Oct 31 00:54:40.432681 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:40.432646    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb" event={"ID":"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc","Type":"ContainerStarted","Data":"8dc0794e1f6ce9df992cc86bac432abe13b2c1c26791e84d4ced53ccb6c6d30e"}
Oct 31 00:54:40.434220 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:40.434201    4935 generic.go:356] "Generic (PLEG): container finished" podID="853937a2-9ba9-4c14-a25d-90ecd6a8a08a" containerID="a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387" exitCode=0
Oct 31 00:54:40.434312 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:40.434269    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk" event={"ID":"853937a2-9ba9-4c14-a25d-90ecd6a8a08a","Type":"ContainerDied","Data":"a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387"}
Oct 31 00:54:40.523513 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:40.523485    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:40.631982 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:40.631921    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb" podStartSLOduration=2.331609432 podStartE2EDuration="4.631906238s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.300296806s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:40.59673221 +0000 UTC m=+1552.779261504" watchObservedRunningTime="2026-10-31 00:54:40.631906238 +0000 UTC m=+1552.814435550"
Oct 31 00:54:41.441209 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:41.441170    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk" event={"ID":"853937a2-9ba9-4c14-a25d-90ecd6a8a08a","Type":"ContainerStarted","Data":"0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a"}
Oct 31 00:54:41.441607 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:41.441529    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:54:41.558872 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:41.558811    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk" podStartSLOduration=3.269916407 podStartE2EDuration="5.558792001s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.288875594s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:41.5555463 +0000 UTC m=+1553.738075594" watchObservedRunningTime="2026-10-31 00:54:41.558792001 +0000 UTC m=+1553.741321298"
Oct 31 00:54:44.152675 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:54:44.152631    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:54:46.522821 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:46.522783    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:46.535269 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:46.535243    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:47.484385 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:47.484360    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:54:50.856630 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:50.856557    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main" containerID="cri-o://64b32714632a03699f6b5e85b5e74202a1ac981a3405bf7333eace80de8ca926" gracePeriod=45
Oct 31 00:54:51.498466 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.498430    4935 generic.go:356] "Generic (PLEG): container finished" podID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerID="64b32714632a03699f6b5e85b5e74202a1ac981a3405bf7333eace80de8ca926" exitCode=0
Oct 31 00:54:51.498671 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.498466    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" event={"ID":"e1d7685f-c2ee-421a-9f4b-7340f478ed20","Type":"ContainerDied","Data":"64b32714632a03699f6b5e85b5e74202a1ac981a3405bf7333eace80de8ca926"}
Oct 31 00:54:51.625769 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.625744    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:54:51.699899 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.699827    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d7685f-c2ee-421a-9f4b-7340f478ed20-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1d7685f-c2ee-421a-9f4b-7340f478ed20-cabundle-cert\") pod \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") "
Oct 31 00:54:51.699899 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.699880    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tmp-dir\") pod \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") "
Oct 31 00:54:51.700148 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.699909    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-model-cache\") pod \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") "
Oct 31 00:54:51.700148 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.699929    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-home\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-home\") pod \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") "
Oct 31 00:54:51.700148 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.699960    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tls-certs\") pod \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") "
Oct 31 00:54:51.700148 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.700012    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-dshm\") pod \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") "
Oct 31 00:54:51.700148 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.700078    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kserve-provision-location\") pod \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") "
Oct 31 00:54:51.700148 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.700110    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kube-api-access-2th2p\" (UniqueName: \"kubernetes.io/projected/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kube-api-access-2th2p\") pod \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\" (UID: \"e1d7685f-c2ee-421a-9f4b-7340f478ed20\") "
Oct 31 00:54:51.700444 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.700214    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-model-cache" pod "e1d7685f-c2ee-421a-9f4b-7340f478ed20" (UID: "e1d7685f-c2ee-421a-9f4b-7340f478ed20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:51.700444 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.700273    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d7685f-c2ee-421a-9f4b-7340f478ed20-cabundle-cert" pod "e1d7685f-c2ee-421a-9f4b-7340f478ed20" (UID: "e1d7685f-c2ee-421a-9f4b-7340f478ed20"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:51.700554 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.700520    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1d7685f-c2ee-421a-9f4b-7340f478ed20-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:51.700554 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.700545    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:51.701446 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.701414    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-home" pod "e1d7685f-c2ee-421a-9f4b-7340f478ed20" (UID: "e1d7685f-c2ee-421a-9f4b-7340f478ed20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:51.702988 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.702955    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kube-api-access-2th2p" pod "e1d7685f-c2ee-421a-9f4b-7340f478ed20" (UID: "e1d7685f-c2ee-421a-9f4b-7340f478ed20"). InnerVolumeSpecName "kube-api-access-2th2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:51.703149 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.703101    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-dshm" pod "e1d7685f-c2ee-421a-9f4b-7340f478ed20" (UID: "e1d7685f-c2ee-421a-9f4b-7340f478ed20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:51.703149 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.703118    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tls-certs" pod "e1d7685f-c2ee-421a-9f4b-7340f478ed20" (UID: "e1d7685f-c2ee-421a-9f4b-7340f478ed20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:54:51.719511 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.719436    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tmp-dir" pod "e1d7685f-c2ee-421a-9f4b-7340f478ed20" (UID: "e1d7685f-c2ee-421a-9f4b-7340f478ed20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:51.728913 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.728875    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kserve-provision-location" pod "e1d7685f-c2ee-421a-9f4b-7340f478ed20" (UID: "e1d7685f-c2ee-421a-9f4b-7340f478ed20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:54:51.801248 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.801214    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:51.801248 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.801242    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:51.801248 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.801254    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d7685f-c2ee-421a-9f4b-7340f478ed20-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:51.801468 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.801264    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:51.801468 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.801277    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:51.801468 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:51.801289    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2th2p\" (UniqueName: \"kubernetes.io/projected/e1d7685f-c2ee-421a-9f4b-7340f478ed20-kube-api-access-2th2p\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:52.505812 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:52.505775    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"
Oct 31 00:54:52.505812 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:52.505804    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls" event={"ID":"e1d7685f-c2ee-421a-9f4b-7340f478ed20","Type":"ContainerDied","Data":"a0239d31d295506c7e5691b35c3dc4d481f1184a1e1b349f7f2593502c9c56d2"}
Oct 31 00:54:52.506479 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:52.505867    4935 scope.go:122] "RemoveContainer" containerID="64b32714632a03699f6b5e85b5e74202a1ac981a3405bf7333eace80de8ca926"
Oct 31 00:54:52.516419 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:52.516395    4935 scope.go:122] "RemoveContainer" containerID="56083a7bc4a0622a0df864dba6a52e7728d13e0e84bde83d96d5a99d09d8951c"
Oct 31 00:54:52.714090 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:52.714054    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"]
Oct 31 00:54:52.761189 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:52.761111    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-dd856666c-dpwls"]
Oct 31 00:54:53.943237 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:54:53.943198    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" path="/var/lib/kubelet/pods/e1d7685f-c2ee-421a-9f4b-7340f478ed20/volumes"
Oct 31 00:55:12.457442 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:12.457410    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:55:14.151449 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:55:14.151401    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:55:17.772578 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:17.772547    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk_853937a2-9ba9-4c14-a25d-90ecd6a8a08a/main/0.log"
Oct 31 00:55:21.236378 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.236340    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"]
Oct 31 00:55:21.236830 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.236595    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk" podUID="853937a2-9ba9-4c14-a25d-90ecd6a8a08a" containerName="main" containerID="cri-o://0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a" gracePeriod=30
Oct 31 00:55:21.282093 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.282059    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"]
Oct 31 00:55:21.292502 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:55:21.292468    4935 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="8dc0794e1f6ce9df992cc86bac432abe13b2c1c26791e84d4ced53ccb6c6d30e" cmd=["/bin/sleep","15"]
Oct 31 00:55:21.292636 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.292525    4935 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:55:21.292636 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:55:21.292545    4935 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb" podUID="1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" containerName="main" containerID="cri-o://8dc0794e1f6ce9df992cc86bac432abe13b2c1c26791e84d4ced53ccb6c6d30e"
Oct 31 00:55:21.292636 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.292584    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb" podUID="1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" containerName="main" containerID="cri-o://8dc0794e1f6ce9df992cc86bac432abe13b2c1c26791e84d4ced53ccb6c6d30e" gracePeriod=60
Oct 31 00:55:21.473837 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.473796    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb" podUID="1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 31 00:55:21.648469 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.648437    4935 generic.go:356] "Generic (PLEG): container finished" podID="1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" containerID="8dc0794e1f6ce9df992cc86bac432abe13b2c1c26791e84d4ced53ccb6c6d30e" exitCode=0
Oct 31 00:55:21.648598 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.648513    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb" event={"ID":"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc","Type":"ContainerDied","Data":"8dc0794e1f6ce9df992cc86bac432abe13b2c1c26791e84d4ced53ccb6c6d30e"}
Oct 31 00:55:21.687123 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.687097    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:55:21.787277 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787246    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-cabundle-cert\") pod \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") "
Oct 31 00:55:21.787423 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787299    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-dshm\") pod \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") "
Oct 31 00:55:21.787423 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787322    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kube-api-access-6p5qw\" (UniqueName: \"kubernetes.io/projected/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kube-api-access-6p5qw\") pod \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") "
Oct 31 00:55:21.787423 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787365    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-home\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-home\") pod \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") "
Oct 31 00:55:21.787423 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787390    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tmp-dir\") pod \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") "
Oct 31 00:55:21.787637 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787445    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tls-certs\") pod \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") "
Oct 31 00:55:21.787637 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787485    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kserve-provision-location\") pod \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") "
Oct 31 00:55:21.787637 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787550    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-model-cache\") pod \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\" (UID: \"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc\") "
Oct 31 00:55:21.787783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787658    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tmp-dir" pod "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" (UID: "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:21.787783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787671    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-home" pod "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" (UID: "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:21.787783 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787665    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-cabundle-cert" pod "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" (UID: "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:55:21.787933 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787892    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.787933 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787896    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-model-cache" pod "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" (UID: "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:21.787933 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787929    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.788099 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.787944    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.789682 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.789653    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-dshm" pod "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" (UID: "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:21.789806 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.789751    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kube-api-access-6p5qw" pod "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" (UID: "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc"). InnerVolumeSpecName "kube-api-access-6p5qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:55:21.789927 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.789906    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tls-certs" pod "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" (UID: "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:55:21.817598 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.817567    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kserve-provision-location" pod "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" (UID: "1e3726aa-12d4-4321-88d2-b2f77b6ff1fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:21.817981 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.817960    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:55:21.889207 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.889176    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.889207 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.889201    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.889207 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.889211    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p5qw\" (UniqueName: \"kubernetes.io/projected/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kube-api-access-6p5qw\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.889425 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.889221    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.889425 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.889230    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.989733 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.989695    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-kserve-provision-location\") pod \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") "
Oct 31 00:55:21.989894 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.989810    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tmp-dir\") pod \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") "
Oct 31 00:55:21.989894 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.989831    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tls-certs\" (UniqueName: \"kubernetes.io/secret/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tls-certs\") pod \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") "
Oct 31 00:55:21.989894 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.989863    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-cabundle-cert\") pod \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\" (UID: \"853937a2-9ba9-4c14-a25d-90ecd6a8a08a\") "
Oct 31 00:55:21.990310 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.990289    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tmp-dir" pod "853937a2-9ba9-4c14-a25d-90ecd6a8a08a" (UID: "853937a2-9ba9-4c14-a25d-90ecd6a8a08a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:21.990379 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.990356    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-cabundle-cert" pod "853937a2-9ba9-4c14-a25d-90ecd6a8a08a" (UID: "853937a2-9ba9-4c14-a25d-90ecd6a8a08a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:55:21.992154 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:21.992137    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tls-certs" pod "853937a2-9ba9-4c14-a25d-90ecd6a8a08a" (UID: "853937a2-9ba9-4c14-a25d-90ecd6a8a08a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:55:22.011910 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.011881    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-kserve-provision-location" pod "853937a2-9ba9-4c14-a25d-90ecd6a8a08a" (UID: "853937a2-9ba9-4c14-a25d-90ecd6a8a08a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:22.091479 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.091409    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:22.091479 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.091434    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:22.091479 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.091445    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:22.091479 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.091454    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/853937a2-9ba9-4c14-a25d-90ecd6a8a08a-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:22.655274 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.655237    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb" event={"ID":"1e3726aa-12d4-4321-88d2-b2f77b6ff1fc","Type":"ContainerDied","Data":"d6919a569433ebbd4b8972e04123f1b622e3eb9f75c834ded33b0258a758b41a"}
Oct 31 00:55:22.655274 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.655268    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"
Oct 31 00:55:22.655803 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.655333    4935 scope.go:122] "RemoveContainer" containerID="8dc0794e1f6ce9df992cc86bac432abe13b2c1c26791e84d4ced53ccb6c6d30e"
Oct 31 00:55:22.657019 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.656961    4935 generic.go:356] "Generic (PLEG): container finished" podID="853937a2-9ba9-4c14-a25d-90ecd6a8a08a" containerID="0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a" exitCode=0
Oct 31 00:55:22.657019 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.656994    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk" event={"ID":"853937a2-9ba9-4c14-a25d-90ecd6a8a08a","Type":"ContainerDied","Data":"0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a"}
Oct 31 00:55:22.657160 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.657023    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"
Oct 31 00:55:22.657160 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.657064    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk" event={"ID":"853937a2-9ba9-4c14-a25d-90ecd6a8a08a","Type":"ContainerDied","Data":"4ea7efe3b5e08df1b152284fb805b80e3e8fb28d21c9a6f118b3ca511d365d7e"}
Oct 31 00:55:22.664953 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.664930    4935 scope.go:122] "RemoveContainer" containerID="bfec38cab9963d545e0bab0f06231a6e499e2790ca7007c315b6262229c0fe6c"
Oct 31 00:55:22.672623 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.672604    4935 scope.go:122] "RemoveContainer" containerID="0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a"
Oct 31 00:55:22.679806 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.679792    4935 scope.go:122] "RemoveContainer" containerID="a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387"
Oct 31 00:55:22.686987 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.686968    4935 scope.go:122] "RemoveContainer" containerID="0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a"
Oct 31 00:55:22.687267 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:55:22.687249    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a\": container with ID starting with 0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a not found: ID does not exist" containerID="0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a"
Oct 31 00:55:22.687331 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.687275    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a"} err="failed to get container status \"0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a\": rpc error: code = NotFound desc = could not find container \"0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a\": container with ID starting with 0b99ca3b9e37cd8cf0065e905aad4778b447fb2b5aaf532ddbb0466a0348c04a not found: ID does not exist"
Oct 31 00:55:22.687331 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.687291    4935 scope.go:122] "RemoveContainer" containerID="a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387"
Oct 31 00:55:22.687512 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:55:22.687487    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387\": container with ID starting with a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387 not found: ID does not exist" containerID="a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387"
Oct 31 00:55:22.687556 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.687512    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387"} err="failed to get container status \"a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387\": rpc error: code = NotFound desc = could not find container \"a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387\": container with ID starting with a260a73e4548302b90d82561866cc9b0877edc956a79cbaed7732cdee1e7a387 not found: ID does not exist"
Oct 31 00:55:22.864609 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.864577    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"]
Oct 31 00:55:22.911298 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.911230    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-gqhwb"]
Oct 31 00:55:22.979566 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.979533    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"]
Oct 31 00:55:22.988615 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:22.988587    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-74bfcc68d9-csljk"]
Oct 31 00:55:23.943592 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:23.943558    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" path="/var/lib/kubelet/pods/1e3726aa-12d4-4321-88d2-b2f77b6ff1fc/volumes"
Oct 31 00:55:23.944003 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:23.943989    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="853937a2-9ba9-4c14-a25d-90ecd6a8a08a" path="/var/lib/kubelet/pods/853937a2-9ba9-4c14-a25d-90ecd6a8a08a/volumes"
Oct 31 00:55:28.620623 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.620583    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"]
Oct 31 00:55:28.621195 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621176    4935 cpu_manager.go:414] "containerMap: removing container" podUID="853937a2-9ba9-4c14-a25d-90ecd6a8a08a" containerName="main"
Oct 31 00:55:28.621284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621197    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853937a2-9ba9-4c14-a25d-90ecd6a8a08a" containerName="main"
Oct 31 00:55:28.621284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621219    4935 cpu_manager.go:414] "containerMap: removing container" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="storage-initializer"
Oct 31 00:55:28.621284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621225    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="storage-initializer"
Oct 31 00:55:28.621284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621238    4935 cpu_manager.go:414] "containerMap: removing container" podUID="1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" containerName="main"
Oct 31 00:55:28.621284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621245    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" containerName="main"
Oct 31 00:55:28.621284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621264    4935 cpu_manager.go:414] "containerMap: removing container" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main"
Oct 31 00:55:28.621284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621270    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main"
Oct 31 00:55:28.621284 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621285    4935 cpu_manager.go:414] "containerMap: removing container" podUID="853937a2-9ba9-4c14-a25d-90ecd6a8a08a" containerName="storage-initializer"
Oct 31 00:55:28.621654 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621292    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853937a2-9ba9-4c14-a25d-90ecd6a8a08a" containerName="storage-initializer"
Oct 31 00:55:28.621654 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621308    4935 cpu_manager.go:414] "containerMap: removing container" podUID="1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" containerName="storage-initializer"
Oct 31 00:55:28.621654 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621316    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" containerName="storage-initializer"
Oct 31 00:55:28.621654 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621404    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="853937a2-9ba9-4c14-a25d-90ecd6a8a08a" containerName="main"
Oct 31 00:55:28.621654 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621417    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3726aa-12d4-4321-88d2-b2f77b6ff1fc" containerName="main"
Oct 31 00:55:28.621654 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.621429    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d7685f-c2ee-421a-9f4b-7340f478ed20" containerName="main"
Oct 31 00:55:28.626469 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.626446    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.651304 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.651275    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981d44c9-81ea-41be-98d5-0a21f14916a2-cabundle-cert\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.651447 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.651315    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-tmp-dir\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.651447 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.651398    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/981d44c9-81ea-41be-98d5-0a21f14916a2-tls-certs\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.651447 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.651434    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-dshm\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.651555 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.651521    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.651586 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.651556    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-model-cache\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.651624 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.651586    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlb2x\" (UniqueName: \"kubernetes.io/projected/981d44c9-81ea-41be-98d5-0a21f14916a2-kube-api-access-jlb2x\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.651678 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.651634    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-home\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.677289 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.677258    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Oct 31 00:55:28.678264 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.678237    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"]
Oct 31 00:55:28.723194 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.723157    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Oct 31 00:55:28.723359 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.723211    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Oct 31 00:55:28.723359 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.723232    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 00:55:28.723359 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.723250    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Oct 31 00:55:28.723359 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.723273    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-gttt6\""
Oct 31 00:55:28.753068 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753020    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlb2x\" (UniqueName: \"kubernetes.io/projected/981d44c9-81ea-41be-98d5-0a21f14916a2-kube-api-access-jlb2x\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753194 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753076    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-home\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753194 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753108    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981d44c9-81ea-41be-98d5-0a21f14916a2-cabundle-cert\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753194 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753124    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-tmp-dir\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753194 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753144    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/981d44c9-81ea-41be-98d5-0a21f14916a2-tls-certs\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753194 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753166    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-dshm\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753438 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753227    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753438 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753267    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-model-cache\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753610 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753584    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-tmp-dir\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753754 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753609    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-home\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753754 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753688    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753862 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753783    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-model-cache\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.753939 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.753919    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981d44c9-81ea-41be-98d5-0a21f14916a2-cabundle-cert\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.755666 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.755643    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-dshm\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.755876 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.755862    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/981d44c9-81ea-41be-98d5-0a21f14916a2-tls-certs\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.813285 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.813256    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlb2x\" (UniqueName: \"kubernetes.io/projected/981d44c9-81ea-41be-98d5-0a21f14916a2-kube-api-access-jlb2x\") pod \"auth-invalid-token-test-kserve-647fc856f5-6rvmk\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:28.945130 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:28.945014    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:29.372944 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:29.372908    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"]
Oct 31 00:55:29.379980 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:55:29.379950    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981d44c9_81ea_41be_98d5_0a21f14916a2.slice/crio-0ae019070f78b6593333b5704255e9694874f46e9dcad152d8f6204ef91a3297 WatchSource:0}: Error finding container 0ae019070f78b6593333b5704255e9694874f46e9dcad152d8f6204ef91a3297: Status 404 returned error can't find the container with id 0ae019070f78b6593333b5704255e9694874f46e9dcad152d8f6204ef91a3297
Oct 31 00:55:29.695470 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:29.695379    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" event={"ID":"981d44c9-81ea-41be-98d5-0a21f14916a2","Type":"ContainerStarted","Data":"0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec"}
Oct 31 00:55:29.695470 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:29.695440    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" event={"ID":"981d44c9-81ea-41be-98d5-0a21f14916a2","Type":"ContainerStarted","Data":"0ae019070f78b6593333b5704255e9694874f46e9dcad152d8f6204ef91a3297"}
Oct 31 00:55:31.707172 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:31.707134    4935 generic.go:356] "Generic (PLEG): container finished" podID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerID="0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec" exitCode=0
Oct 31 00:55:31.707548 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:31.707205    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" event={"ID":"981d44c9-81ea-41be-98d5-0a21f14916a2","Type":"ContainerDied","Data":"0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec"}
Oct 31 00:55:32.713825 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:32.713788    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" event={"ID":"981d44c9-81ea-41be-98d5-0a21f14916a2","Type":"ContainerStarted","Data":"ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6"}
Oct 31 00:55:32.928434 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:32.928377    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podStartSLOduration=2.861816116 podStartE2EDuration="4.928360723s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.066544607s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:32.926757078 +0000 UTC m=+1605.109286374" watchObservedRunningTime="2026-10-31 00:55:32.928360723 +0000 UTC m=+1605.110890010"
Oct 31 00:55:32.945406 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:32.945368    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:38.946111 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:38.946076    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:55:38.947613 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:38.947586    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 31 00:55:44.157255 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:55:44.157210    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:55:48.946073 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:48.946011    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 31 00:55:58.946496 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:55:58.946454    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 31 00:56:08.945991 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:08.945950    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 31 00:56:14.154672 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:56:14.154632    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:56:16.807681 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.807642    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"]
Oct 31 00:56:16.811349 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.811332    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:16.862723 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.862689    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Oct 31 00:56:16.862900 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.862879    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Oct 31 00:56:16.908891 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.908839    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 00:56:16.908891 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.908862    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-cdn7h\""
Oct 31 00:56:16.908891 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.908862    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Oct 31 00:56:16.908891 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.908840    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Oct 31 00:56:16.909852 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.909814    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68feaa40-b98f-464a-84e6-db602a88ba66-tls-certs\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:16.909983 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.909858    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68feaa40-b98f-464a-84e6-db602a88ba66-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:16.909983 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.909880    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-home\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:16.910141 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.909992    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-model-cache\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:16.910141 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.910069    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5zq\" (UniqueName: \"kubernetes.io/projected/68feaa40-b98f-464a-84e6-db602a88ba66-kube-api-access-rl5zq\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:16.910141 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.910116    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-dshm\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:16.910304 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.910148    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:16.910304 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.910236    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:16.911548 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:16.911523    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"]
Oct 31 00:56:17.011261 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.011226    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68feaa40-b98f-464a-84e6-db602a88ba66-tls-certs\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.011261 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.011266    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68feaa40-b98f-464a-84e6-db602a88ba66-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.011519 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.011285    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-home\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.011519 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.011315    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-model-cache\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.011519 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.011369    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5zq\" (UniqueName: \"kubernetes.io/projected/68feaa40-b98f-464a-84e6-db602a88ba66-kube-api-access-rl5zq\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.011519 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.011393    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-dshm\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.011519 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.011423    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.011519 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.011448    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.011845 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.011707    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-home\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.011845 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.011781    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-model-cache\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.012066 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.012023    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.012163 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.012103    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.012372 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.012349    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68feaa40-b98f-464a-84e6-db602a88ba66-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.013931 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.013913    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-dshm\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.014285 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.014263    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68feaa40-b98f-464a-84e6-db602a88ba66-tls-certs\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.077229 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.077145    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5zq\" (UniqueName: \"kubernetes.io/projected/68feaa40-b98f-464a-84e6-db602a88ba66-kube-api-access-rl5zq\") pod \"tokenizer-template-args-test-kserve-7995846755-fcp5m\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.131886 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.131856    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:17.519341 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.519304    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"]
Oct 31 00:56:17.525078 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.525054    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.559710 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.559537    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"]
Oct 31 00:56:17.567061 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:56:17.567018    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68feaa40_b98f_464a_84e6_db602a88ba66.slice/crio-a321b2e7d96119e181fb072eb7337930e59b457952440dd5d0fa02b9c0424d8c WatchSource:0}: Error finding container a321b2e7d96119e181fb072eb7337930e59b457952440dd5d0fa02b9c0424d8c: Status 404 returned error can't find the container with id a321b2e7d96119e181fb072eb7337930e59b457952440dd5d0fa02b9c0424d8c
Oct 31 00:56:17.581849 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.581822    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"]
Oct 31 00:56:17.617686 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.617661    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74370c27-079f-4db1-9fe9-9c3195f34fbd-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.617807 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.617693    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.617807 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.617716    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74370c27-079f-4db1-9fe9-9c3195f34fbd-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.617939 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.617843    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.718333 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.718302    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74370c27-079f-4db1-9fe9-9c3195f34fbd-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.718333 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.718337    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.718611 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.718359    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74370c27-079f-4db1-9fe9-9c3195f34fbd-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.718611 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.718418    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.718855 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.718838    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.718914 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.718858    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.719126 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.719105    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74370c27-079f-4db1-9fe9-9c3195f34fbd-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.722012 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.721988    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74370c27-079f-4db1-9fe9-9c3195f34fbd-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.841131 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.841024    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:17.946576 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.946536    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m" event={"ID":"68feaa40-b98f-464a-84e6-db602a88ba66","Type":"ContainerStarted","Data":"e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26"}
Oct 31 00:56:17.946780 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:17.946609    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m" event={"ID":"68feaa40-b98f-464a-84e6-db602a88ba66","Type":"ContainerStarted","Data":"a321b2e7d96119e181fb072eb7337930e59b457952440dd5d0fa02b9c0424d8c"}
Oct 31 00:56:18.219850 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:18.219766    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"]
Oct 31 00:56:18.225463 ip-10-0-132-125 kubenswrapper[4935]: W1031 00:56:18.225436    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74370c27_079f_4db1_9fe9_9c3195f34fbd.slice/crio-a9203638fef78fa4bc3fd8ef479e38707171867fb179385bf1ed0494f0dbb150 WatchSource:0}: Error finding container a9203638fef78fa4bc3fd8ef479e38707171867fb179385bf1ed0494f0dbb150: Status 404 returned error can't find the container with id a9203638fef78fa4bc3fd8ef479e38707171867fb179385bf1ed0494f0dbb150
Oct 31 00:56:18.948132 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:18.948094    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 31 00:56:18.958059 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:18.955229    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh" event={"ID":"74370c27-079f-4db1-9fe9-9c3195f34fbd","Type":"ContainerStarted","Data":"1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692"}
Oct 31 00:56:18.958059 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:18.955298    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh" event={"ID":"74370c27-079f-4db1-9fe9-9c3195f34fbd","Type":"ContainerStarted","Data":"a9203638fef78fa4bc3fd8ef479e38707171867fb179385bf1ed0494f0dbb150"}
Oct 31 00:56:20.970801 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:20.970717    4935 generic.go:356] "Generic (PLEG): container finished" podID="74370c27-079f-4db1-9fe9-9c3195f34fbd" containerID="1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692" exitCode=0
Oct 31 00:56:20.971304 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:20.970793    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh" event={"ID":"74370c27-079f-4db1-9fe9-9c3195f34fbd","Type":"ContainerDied","Data":"1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692"}
Oct 31 00:56:20.972392 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:20.972370    4935 generic.go:356] "Generic (PLEG): container finished" podID="68feaa40-b98f-464a-84e6-db602a88ba66" containerID="e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26" exitCode=0
Oct 31 00:56:20.972490 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:20.972445    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m" event={"ID":"68feaa40-b98f-464a-84e6-db602a88ba66","Type":"ContainerDied","Data":"e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26"}
Oct 31 00:56:21.978753 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:21.978714    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m" event={"ID":"68feaa40-b98f-464a-84e6-db602a88ba66","Type":"ContainerStarted","Data":"42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f"}
Oct 31 00:56:21.980807 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:21.980782    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh" event={"ID":"74370c27-079f-4db1-9fe9-9c3195f34fbd","Type":"ContainerStarted","Data":"3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858"}
Oct 31 00:56:21.980994 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:21.980975    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:22.100323 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:22.100259    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m" podStartSLOduration=3.699995912 podStartE2EDuration="6.100241123s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.400245211s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:22.099371237 +0000 UTC m=+1654.281900532" watchObservedRunningTime="2026-10-31 00:56:22.100241123 +0000 UTC m=+1654.282770418"
Oct 31 00:56:22.132776 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:22.132734    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:22.216847 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:22.216783    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh" podStartSLOduration=2.806489713 podStartE2EDuration="5.216764181s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.410274468s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:22.215334685 +0000 UTC m=+1654.397864030" watchObservedRunningTime="2026-10-31 00:56:22.216764181 +0000 UTC m=+1654.399293474"
Oct 31 00:56:27.132880 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:27.132668    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:27.150077 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:27.150019    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:28.025098 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:28.025068    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:56:28.946479 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:28.946427    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 31 00:56:38.946425 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:38.946380    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 31 00:56:44.155744 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:56:44.155701    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:56:48.946409 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:48.946361    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 31 00:56:52.997766 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:52.997732    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:56:56.933430 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:56.933400    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh_74370c27-079f-4db1-9fe9-9c3195f34fbd/main/0.log"
Oct 31 00:56:58.945877 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:56:58.945836    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 31 00:57:00.967844 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:00.967814    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"]
Oct 31 00:57:00.968295 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:00.968050    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh" podUID="74370c27-079f-4db1-9fe9-9c3195f34fbd" containerName="main" containerID="cri-o://3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858" gracePeriod=30
Oct 31 00:57:00.980578 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:00.980546    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"]
Oct 31 00:57:00.990556 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:57:00.990519    4935 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f" cmd=["/bin/sleep","15"]
Oct 31 00:57:00.990688 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:00.990575    4935 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:57:00.990688 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:57:00.990589    4935 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m" podUID="68feaa40-b98f-464a-84e6-db602a88ba66" containerName="main" containerID="cri-o://42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f"
Oct 31 00:57:00.990688 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:00.990628    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m" podUID="68feaa40-b98f-464a-84e6-db602a88ba66" containerName="main" containerID="cri-o://42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f" gracePeriod=60
Oct 31 00:57:01.014532 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.014488    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m" podUID="68feaa40-b98f-464a-84e6-db602a88ba66" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 31 00:57:01.372531 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.372505    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:57:01.482169 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.482137    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68feaa40-b98f-464a-84e6-db602a88ba66-tls-certs\" (UniqueName: \"kubernetes.io/secret/68feaa40-b98f-464a-84e6-db602a88ba66-tls-certs\") pod \"68feaa40-b98f-464a-84e6-db602a88ba66\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") "
Oct 31 00:57:01.482364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.482187    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68feaa40-b98f-464a-84e6-db602a88ba66-kube-api-access-rl5zq\" (UniqueName: \"kubernetes.io/projected/68feaa40-b98f-464a-84e6-db602a88ba66-kube-api-access-rl5zq\") pod \"68feaa40-b98f-464a-84e6-db602a88ba66\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") "
Oct 31 00:57:01.482364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.482238    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68feaa40-b98f-464a-84e6-db602a88ba66-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68feaa40-b98f-464a-84e6-db602a88ba66-cabundle-cert\") pod \"68feaa40-b98f-464a-84e6-db602a88ba66\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") "
Oct 31 00:57:01.482364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.482284    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-dshm\") pod \"68feaa40-b98f-464a-84e6-db602a88ba66\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") "
Oct 31 00:57:01.482364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.482345    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-tmp-dir\") pod \"68feaa40-b98f-464a-84e6-db602a88ba66\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") "
Oct 31 00:57:01.482574 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.482385    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-model-cache\") pod \"68feaa40-b98f-464a-84e6-db602a88ba66\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") "
Oct 31 00:57:01.482574 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.482425    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-kserve-provision-location\") pod \"68feaa40-b98f-464a-84e6-db602a88ba66\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") "
Oct 31 00:57:01.482574 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.482472    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-home\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-home\") pod \"68feaa40-b98f-464a-84e6-db602a88ba66\" (UID: \"68feaa40-b98f-464a-84e6-db602a88ba66\") "
Oct 31 00:57:01.484821 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.483117    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-home" pod "68feaa40-b98f-464a-84e6-db602a88ba66" (UID: "68feaa40-b98f-464a-84e6-db602a88ba66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.484821 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.483353    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-tmp-dir" pod "68feaa40-b98f-464a-84e6-db602a88ba66" (UID: "68feaa40-b98f-464a-84e6-db602a88ba66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.484821 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.483549    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68feaa40-b98f-464a-84e6-db602a88ba66-cabundle-cert" pod "68feaa40-b98f-464a-84e6-db602a88ba66" (UID: "68feaa40-b98f-464a-84e6-db602a88ba66"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:57:01.484821 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.483631    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-model-cache" pod "68feaa40-b98f-464a-84e6-db602a88ba66" (UID: "68feaa40-b98f-464a-84e6-db602a88ba66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.486186 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.486156    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68feaa40-b98f-464a-84e6-db602a88ba66-tls-certs" pod "68feaa40-b98f-464a-84e6-db602a88ba66" (UID: "68feaa40-b98f-464a-84e6-db602a88ba66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:57:01.486309 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.486281    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68feaa40-b98f-464a-84e6-db602a88ba66-kube-api-access-rl5zq" pod "68feaa40-b98f-464a-84e6-db602a88ba66" (UID: "68feaa40-b98f-464a-84e6-db602a88ba66"). InnerVolumeSpecName "kube-api-access-rl5zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:57:01.513371 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.513326    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-dshm" pod "68feaa40-b98f-464a-84e6-db602a88ba66" (UID: "68feaa40-b98f-464a-84e6-db602a88ba66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.515341 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.515304    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-kserve-provision-location" pod "68feaa40-b98f-464a-84e6-db602a88ba66" (UID: "68feaa40-b98f-464a-84e6-db602a88ba66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.584326 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.584288    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.584326 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.584329    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68feaa40-b98f-464a-84e6-db602a88ba66-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.584517 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.584345    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl5zq\" (UniqueName: \"kubernetes.io/projected/68feaa40-b98f-464a-84e6-db602a88ba66-kube-api-access-rl5zq\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.584517 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.584362    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68feaa40-b98f-464a-84e6-db602a88ba66-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.584517 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.584376    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.584517 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.584393    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.584517 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.584407    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.584517 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.584424    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68feaa40-b98f-464a-84e6-db602a88ba66-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.622081 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.622053    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:57:01.684824 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.684795    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-kserve-provision-location\") pod \"74370c27-079f-4db1-9fe9-9c3195f34fbd\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") "
Oct 31 00:57:01.684974 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.684900    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-tmp-dir\") pod \"74370c27-079f-4db1-9fe9-9c3195f34fbd\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") "
Oct 31 00:57:01.684974 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.684951    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74370c27-079f-4db1-9fe9-9c3195f34fbd-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74370c27-079f-4db1-9fe9-9c3195f34fbd-cabundle-cert\") pod \"74370c27-079f-4db1-9fe9-9c3195f34fbd\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") "
Oct 31 00:57:01.685095 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.684973    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74370c27-079f-4db1-9fe9-9c3195f34fbd-tls-certs\" (UniqueName: \"kubernetes.io/secret/74370c27-079f-4db1-9fe9-9c3195f34fbd-tls-certs\") pod \"74370c27-079f-4db1-9fe9-9c3195f34fbd\" (UID: \"74370c27-079f-4db1-9fe9-9c3195f34fbd\") "
Oct 31 00:57:01.685352 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.685322    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74370c27-079f-4db1-9fe9-9c3195f34fbd-cabundle-cert" pod "74370c27-079f-4db1-9fe9-9c3195f34fbd" (UID: "74370c27-079f-4db1-9fe9-9c3195f34fbd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:57:01.685427 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.685362    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-tmp-dir" pod "74370c27-079f-4db1-9fe9-9c3195f34fbd" (UID: "74370c27-079f-4db1-9fe9-9c3195f34fbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.687185 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.687155    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74370c27-079f-4db1-9fe9-9c3195f34fbd-tls-certs" pod "74370c27-079f-4db1-9fe9-9c3195f34fbd" (UID: "74370c27-079f-4db1-9fe9-9c3195f34fbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:57:01.715746 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.715712    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-kserve-provision-location" pod "74370c27-079f-4db1-9fe9-9c3195f34fbd" (UID: "74370c27-079f-4db1-9fe9-9c3195f34fbd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.786352 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.786320    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.786352 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.786349    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74370c27-079f-4db1-9fe9-9c3195f34fbd-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.786352 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.786360    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74370c27-079f-4db1-9fe9-9c3195f34fbd-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.786594 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:01.786370    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74370c27-079f-4db1-9fe9-9c3195f34fbd-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:02.204478 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.204380    4935 generic.go:356] "Generic (PLEG): container finished" podID="68feaa40-b98f-464a-84e6-db602a88ba66" containerID="42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f" exitCode=0
Oct 31 00:57:02.204478 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.204437    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"
Oct 31 00:57:02.204478 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.204465    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m" event={"ID":"68feaa40-b98f-464a-84e6-db602a88ba66","Type":"ContainerDied","Data":"42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f"}
Oct 31 00:57:02.205153 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.204522    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m" event={"ID":"68feaa40-b98f-464a-84e6-db602a88ba66","Type":"ContainerDied","Data":"a321b2e7d96119e181fb072eb7337930e59b457952440dd5d0fa02b9c0424d8c"}
Oct 31 00:57:02.205153 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.204552    4935 scope.go:122] "RemoveContainer" containerID="42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f"
Oct 31 00:57:02.206541 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.206508    4935 generic.go:356] "Generic (PLEG): container finished" podID="74370c27-079f-4db1-9fe9-9c3195f34fbd" containerID="3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858" exitCode=0
Oct 31 00:57:02.206638 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.206562    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"
Oct 31 00:57:02.206703 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.206555    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh" event={"ID":"74370c27-079f-4db1-9fe9-9c3195f34fbd","Type":"ContainerDied","Data":"3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858"}
Oct 31 00:57:02.206703 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.206685    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh" event={"ID":"74370c27-079f-4db1-9fe9-9c3195f34fbd","Type":"ContainerDied","Data":"a9203638fef78fa4bc3fd8ef479e38707171867fb179385bf1ed0494f0dbb150"}
Oct 31 00:57:02.214672 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.214653    4935 scope.go:122] "RemoveContainer" containerID="e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26"
Oct 31 00:57:02.223184 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.223157    4935 scope.go:122] "RemoveContainer" containerID="42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f"
Oct 31 00:57:02.223468 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:57:02.223444    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f\": container with ID starting with 42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f not found: ID does not exist" containerID="42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f"
Oct 31 00:57:02.223517 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.223476    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f"} err="failed to get container status \"42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f\": rpc error: code = NotFound desc = could not find container \"42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f\": container with ID starting with 42bf41950850f0c871d8cb95ab175f48849f5a725f3d5ef71fe0fd70aa8f0d0f not found: ID does not exist"
Oct 31 00:57:02.223517 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.223492    4935 scope.go:122] "RemoveContainer" containerID="e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26"
Oct 31 00:57:02.223761 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:57:02.223737    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26\": container with ID starting with e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26 not found: ID does not exist" containerID="e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26"
Oct 31 00:57:02.223822 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.223772    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26"} err="failed to get container status \"e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26\": rpc error: code = NotFound desc = could not find container \"e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26\": container with ID starting with e3b6fc2d2409196c8ac59f3cb7b9df542b0811c2fe391533a87746ee61678a26 not found: ID does not exist"
Oct 31 00:57:02.223822 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.223797    4935 scope.go:122] "RemoveContainer" containerID="3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858"
Oct 31 00:57:02.231860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.231845    4935 scope.go:122] "RemoveContainer" containerID="1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692"
Oct 31 00:57:02.239375 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.239358    4935 scope.go:122] "RemoveContainer" containerID="3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858"
Oct 31 00:57:02.239643 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:57:02.239614    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858\": container with ID starting with 3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858 not found: ID does not exist" containerID="3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858"
Oct 31 00:57:02.239730 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.239653    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858"} err="failed to get container status \"3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858\": rpc error: code = NotFound desc = could not find container \"3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858\": container with ID starting with 3ff81f432f2fc8b6af1eb2460a3480968c0d8072b6060bf37dc496bfc199f858 not found: ID does not exist"
Oct 31 00:57:02.239730 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.239675    4935 scope.go:122] "RemoveContainer" containerID="1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692"
Oct 31 00:57:02.239925 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:57:02.239906    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692\": container with ID starting with 1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692 not found: ID does not exist" containerID="1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692"
Oct 31 00:57:02.239968 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.239930    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692"} err="failed to get container status \"1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692\": rpc error: code = NotFound desc = could not find container \"1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692\": container with ID starting with 1ba7bc28d95744c82a3812a94206651f7638bfefd6613da2d8500a4671f1c692 not found: ID does not exist"
Oct 31 00:57:02.363568 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.363535    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"]
Oct 31 00:57:02.410621 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.410590    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7d97889c88-5gxvh"]
Oct 31 00:57:02.517208 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.517175    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"]
Oct 31 00:57:02.523153 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:02.523120    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fcp5m"]
Oct 31 00:57:03.943681 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:03.943643    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68feaa40-b98f-464a-84e6-db602a88ba66" path="/var/lib/kubelet/pods/68feaa40-b98f-464a-84e6-db602a88ba66/volumes"
Oct 31 00:57:03.944215 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:03.944083    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74370c27-079f-4db1-9fe9-9c3195f34fbd" path="/var/lib/kubelet/pods/74370c27-079f-4db1-9fe9-9c3195f34fbd/volumes"
Oct 31 00:57:08.945560 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:08.945517    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 31 00:57:14.152164 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:57:14.152126    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:57:18.955254 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:18.955224    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:57:18.962879 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:18.962854    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:57:26.804318 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:26.804284    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"]
Oct 31 00:57:41.827455 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:41.827418    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" containerID="cri-o://ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6" gracePeriod=45
Oct 31 00:57:41.956627 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:41.956588    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 31 00:57:42.587204 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.587152    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:57:42.651948 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.651921    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-kserve-provision-location\") pod \"981d44c9-81ea-41be-98d5-0a21f14916a2\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") "
Oct 31 00:57:42.652093 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.652004    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-home\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-home\") pod \"981d44c9-81ea-41be-98d5-0a21f14916a2\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") "
Oct 31 00:57:42.652093 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.652029    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-tmp-dir\") pod \"981d44c9-81ea-41be-98d5-0a21f14916a2\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") "
Oct 31 00:57:42.652093 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.652067    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981d44c9-81ea-41be-98d5-0a21f14916a2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981d44c9-81ea-41be-98d5-0a21f14916a2-cabundle-cert\") pod \"981d44c9-81ea-41be-98d5-0a21f14916a2\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") "
Oct 31 00:57:42.652259 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.652097    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-model-cache\") pod \"981d44c9-81ea-41be-98d5-0a21f14916a2\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") "
Oct 31 00:57:42.652259 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.652136    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981d44c9-81ea-41be-98d5-0a21f14916a2-kube-api-access-jlb2x\" (UniqueName: \"kubernetes.io/projected/981d44c9-81ea-41be-98d5-0a21f14916a2-kube-api-access-jlb2x\") pod \"981d44c9-81ea-41be-98d5-0a21f14916a2\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") "
Oct 31 00:57:42.652259 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.652164    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/981d44c9-81ea-41be-98d5-0a21f14916a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/981d44c9-81ea-41be-98d5-0a21f14916a2-tls-certs\") pod \"981d44c9-81ea-41be-98d5-0a21f14916a2\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") "
Oct 31 00:57:42.652259 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.652186    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-dshm\") pod \"981d44c9-81ea-41be-98d5-0a21f14916a2\" (UID: \"981d44c9-81ea-41be-98d5-0a21f14916a2\") "
Oct 31 00:57:42.652530 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.652483    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-model-cache" pod "981d44c9-81ea-41be-98d5-0a21f14916a2" (UID: "981d44c9-81ea-41be-98d5-0a21f14916a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:42.652595 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.652533    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981d44c9-81ea-41be-98d5-0a21f14916a2-cabundle-cert" pod "981d44c9-81ea-41be-98d5-0a21f14916a2" (UID: "981d44c9-81ea-41be-98d5-0a21f14916a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:57:42.652744 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.652722    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-home" pod "981d44c9-81ea-41be-98d5-0a21f14916a2" (UID: "981d44c9-81ea-41be-98d5-0a21f14916a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:42.654649 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.654624    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981d44c9-81ea-41be-98d5-0a21f14916a2-tls-certs" pod "981d44c9-81ea-41be-98d5-0a21f14916a2" (UID: "981d44c9-81ea-41be-98d5-0a21f14916a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:57:42.654832 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.654809    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-dshm" pod "981d44c9-81ea-41be-98d5-0a21f14916a2" (UID: "981d44c9-81ea-41be-98d5-0a21f14916a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:42.654916 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.654845    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981d44c9-81ea-41be-98d5-0a21f14916a2-kube-api-access-jlb2x" pod "981d44c9-81ea-41be-98d5-0a21f14916a2" (UID: "981d44c9-81ea-41be-98d5-0a21f14916a2"). InnerVolumeSpecName "kube-api-access-jlb2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:57:42.667133 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.667105    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-tmp-dir" pod "981d44c9-81ea-41be-98d5-0a21f14916a2" (UID: "981d44c9-81ea-41be-98d5-0a21f14916a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:42.677246 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.677222    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-kserve-provision-location" pod "981d44c9-81ea-41be-98d5-0a21f14916a2" (UID: "981d44c9-81ea-41be-98d5-0a21f14916a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:42.752934 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.752904    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981d44c9-81ea-41be-98d5-0a21f14916a2-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:42.753097 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.752937    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:42.753097 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.752953    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlb2x\" (UniqueName: \"kubernetes.io/projected/981d44c9-81ea-41be-98d5-0a21f14916a2-kube-api-access-jlb2x\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:42.753097 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.752967    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/981d44c9-81ea-41be-98d5-0a21f14916a2-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:42.753097 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.752979    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:42.753097 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.752991    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:42.753097 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.753005    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:42.753097 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:42.753018    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/981d44c9-81ea-41be-98d5-0a21f14916a2-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:43.420645 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.420611    4935 generic.go:356] "Generic (PLEG): container finished" podID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerID="ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6" exitCode=0
Oct 31 00:57:43.421207 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.420655    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" event={"ID":"981d44c9-81ea-41be-98d5-0a21f14916a2","Type":"ContainerDied","Data":"ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6"}
Oct 31 00:57:43.421207 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.420669    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"
Oct 31 00:57:43.421207 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.420712    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk" event={"ID":"981d44c9-81ea-41be-98d5-0a21f14916a2","Type":"ContainerDied","Data":"0ae019070f78b6593333b5704255e9694874f46e9dcad152d8f6204ef91a3297"}
Oct 31 00:57:43.421207 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.420737    4935 scope.go:122] "RemoveContainer" containerID="ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6"
Oct 31 00:57:43.429831 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.429815    4935 scope.go:122] "RemoveContainer" containerID="0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec"
Oct 31 00:57:43.437280 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.437265    4935 scope.go:122] "RemoveContainer" containerID="ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6"
Oct 31 00:57:43.437548 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:57:43.437529    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6\": container with ID starting with ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6 not found: ID does not exist" containerID="ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6"
Oct 31 00:57:43.437596 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.437555    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6"} err="failed to get container status \"ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6\": rpc error: code = NotFound desc = could not find container \"ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6\": container with ID starting with ffb8e240b30ee2b87be80cb80e6090cfbaa96d0de8acc3100c00da0e58642af6 not found: ID does not exist"
Oct 31 00:57:43.437596 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.437574    4935 scope.go:122] "RemoveContainer" containerID="0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec"
Oct 31 00:57:43.437772 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:57:43.437757    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec\": container with ID starting with 0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec not found: ID does not exist" containerID="0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec"
Oct 31 00:57:43.437809 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.437775    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec"} err="failed to get container status \"0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec\": rpc error: code = NotFound desc = could not find container \"0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec\": container with ID starting with 0b92cafcc2c62e0a6fca14c84c22513eb791ec824be3c3ba36e6e2f0fbb54bec not found: ID does not exist"
Oct 31 00:57:43.632117 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.632082    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"]
Oct 31 00:57:43.677089 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.677015    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-647fc856f5-6rvmk"]
Oct 31 00:57:43.943626 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:57:43.943555    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" path="/var/lib/kubelet/pods/981d44c9-81ea-41be-98d5-0a21f14916a2/volumes"
Oct 31 00:57:44.152560 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:57:44.152524    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:58:14.152235 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:58:14.152194    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:58:23.550312 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:23.550279    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"]
Oct 31 00:58:23.550817 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:23.550483    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy" containerID="cri-o://913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c" gracePeriod=30
Oct 31 00:58:23.597399 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:23.597368    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"]
Oct 31 00:58:23.597639 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:23.597594    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy" containerID="cri-o://b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654" gracePeriod=30
Oct 31 00:58:28.846475 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.846444    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:58:28.873540 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.873516    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:58:28.936711 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.936680    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-socket\") pod \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") "
Oct 31 00:58:28.936858 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.936733    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-kube-api-access-w496w\" (UniqueName: \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-kube-api-access-w496w\") pod \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") "
Oct 31 00:58:28.936858 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.936795    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-token\" (UniqueName: \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-token\") pod \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") "
Oct 31 00:58:28.936858 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.936825    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istiod-ca-cert\") pod \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") "
Oct 31 00:58:28.937027 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.936909    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-socket" pod "3f48454a-47d5-4f0b-b215-c64be8f3baa4" (UID: "3f48454a-47d5-4f0b-b215-c64be8f3baa4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:58:28.937027 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.936962    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-envoy\") pod \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") "
Oct 31 00:58:28.937027 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937021    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-credential-socket\") pod \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") "
Oct 31 00:58:28.937222 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937115    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-data\") pod \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") "
Oct 31 00:58:28.937276 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937256    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istiod-ca-cert" pod "3f48454a-47d5-4f0b-b215-c64be8f3baa4" (UID: "3f48454a-47d5-4f0b-b215-c64be8f3baa4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:58:28.937351 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937325    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-certs\") pod \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") "
Oct 31 00:58:28.937351 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937329    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-credential-socket" pod "3f48454a-47d5-4f0b-b215-c64be8f3baa4" (UID: "3f48454a-47d5-4f0b-b215-c64be8f3baa4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:58:28.937490 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937367    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-podinfo\") pod \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\" (UID: \"3f48454a-47d5-4f0b-b215-c64be8f3baa4\") "
Oct 31 00:58:28.937580 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937551    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-data" pod "3f48454a-47d5-4f0b-b215-c64be8f3baa4" (UID: "3f48454a-47d5-4f0b-b215-c64be8f3baa4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:58:28.937813 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937590    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-certs" pod "3f48454a-47d5-4f0b-b215-c64be8f3baa4" (UID: "3f48454a-47d5-4f0b-b215-c64be8f3baa4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:58:28.937813 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937800    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:28.937813 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937814    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:28.937990 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937823    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:28.937990 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937831    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:28.937990 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.937839    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:28.939108 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.939090    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-kube-api-access-w496w" pod "3f48454a-47d5-4f0b-b215-c64be8f3baa4" (UID: "3f48454a-47d5-4f0b-b215-c64be8f3baa4"). InnerVolumeSpecName "kube-api-access-w496w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:58:28.939313 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.939297    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-envoy" pod "3f48454a-47d5-4f0b-b215-c64be8f3baa4" (UID: "3f48454a-47d5-4f0b-b215-c64be8f3baa4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:58:28.939617 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.939602    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-podinfo" pod "3f48454a-47d5-4f0b-b215-c64be8f3baa4" (UID: "3f48454a-47d5-4f0b-b215-c64be8f3baa4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:58:28.939665 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:28.939601    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-token" pod "3f48454a-47d5-4f0b-b215-c64be8f3baa4" (UID: "3f48454a-47d5-4f0b-b215-c64be8f3baa4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:58:29.038890 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.038863    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-kube-api-access-tjs4x\" (UniqueName: \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-kube-api-access-tjs4x\") pod \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") "
Oct 31 00:58:29.039071 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.038904    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-data\") pod \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") "
Oct 31 00:58:29.039071 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.038939    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istiod-ca-cert\") pod \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") "
Oct 31 00:58:29.039071 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.038970    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-token\" (UniqueName: \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-token\") pod \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") "
Oct 31 00:58:29.039071 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039025    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-podinfo\") pod \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") "
Oct 31 00:58:29.039348 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039092    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-socket\") pod \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") "
Oct 31 00:58:29.039348 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039151    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-credential-socket\") pod \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") "
Oct 31 00:58:29.039348 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039181    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-certs\") pod \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") "
Oct 31 00:58:29.039348 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039227    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-envoy\") pod \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\" (UID: \"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105\") "
Oct 31 00:58:29.039348 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039240    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-data" pod "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" (UID: "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039455    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-credential-socket" pod "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" (UID: "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039493    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istiod-ca-cert" pod "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" (UID: "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039502    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-certs" pod "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" (UID: "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039594    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039612    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039667    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039681    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039693    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f48454a-47d5-4f0b-b215-c64be8f3baa4-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039705    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039717    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w496w\" (UniqueName: \"kubernetes.io/projected/3f48454a-47d5-4f0b-b215-c64be8f3baa4-kube-api-access-w496w\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039732    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.039860 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.039749    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-socket" pod "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" (UID: "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:58:29.041689 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.041669    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-envoy" pod "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" (UID: "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:58:29.041903 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.041881    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-token" pod "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" (UID: "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:58:29.041968 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.041886    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-podinfo" pod "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" (UID: "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:58:29.042013 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.041968    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-kube-api-access-tjs4x" pod "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" (UID: "1d45ed09-bb56-4f2e-aa5e-1aceab5ce105"). InnerVolumeSpecName "kube-api-access-tjs4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:58:29.140458 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.140379    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.140458 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.140403    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.140458 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.140414    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjs4x\" (UniqueName: \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-kube-api-access-tjs4x\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.140458 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.140423    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.140458 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.140431    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:29.632942 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.632909    4935 generic.go:356] "Generic (PLEG): container finished" podID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerID="b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654" exitCode=0
Oct 31 00:58:29.633107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.632961    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"
Oct 31 00:58:29.633107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.632992    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" event={"ID":"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105","Type":"ContainerDied","Data":"b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654"}
Oct 31 00:58:29.633107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.633060    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz" event={"ID":"1d45ed09-bb56-4f2e-aa5e-1aceab5ce105","Type":"ContainerDied","Data":"4552934c33a3e499e580d4dac7b05d7d491d4ebec92be652e1fdb01a5da358d7"}
Oct 31 00:58:29.633107 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.633090    4935 scope.go:122] "RemoveContainer" containerID="b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654"
Oct 31 00:58:29.634897 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.634870    4935 generic.go:356] "Generic (PLEG): container finished" podID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerID="913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c" exitCode=0
Oct 31 00:58:29.635013 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.634920    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"
Oct 31 00:58:29.635013 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.634926    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" event={"ID":"3f48454a-47d5-4f0b-b215-c64be8f3baa4","Type":"ContainerDied","Data":"913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c"}
Oct 31 00:58:29.635013 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.634968    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2" event={"ID":"3f48454a-47d5-4f0b-b215-c64be8f3baa4","Type":"ContainerDied","Data":"df3bdfcf02e17263c00e6ecee840fe0337286a6b15a43380387a4f6cced3f7cc"}
Oct 31 00:58:29.643013 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.642993    4935 scope.go:122] "RemoveContainer" containerID="dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf"
Oct 31 00:58:29.653815 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.653800    4935 scope.go:122] "RemoveContainer" containerID="b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654"
Oct 31 00:58:29.654082 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:58:29.654065    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654\": container with ID starting with b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654 not found: ID does not exist" containerID="b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654"
Oct 31 00:58:29.654120 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.654092    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654"} err="failed to get container status \"b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654\": rpc error: code = NotFound desc = could not find container \"b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654\": container with ID starting with b4a9d2cd2d3dd85aa089062903b287634da555b2f6195295de9f4c1c08bf1654 not found: ID does not exist"
Oct 31 00:58:29.654120 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.654107    4935 scope.go:122] "RemoveContainer" containerID="dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf"
Oct 31 00:58:29.654315 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:58:29.654302    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf\": container with ID starting with dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf not found: ID does not exist" containerID="dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf"
Oct 31 00:58:29.654355 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.654319    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf"} err="failed to get container status \"dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf\": rpc error: code = NotFound desc = could not find container \"dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf\": container with ID starting with dab18d3ad7f0ce6253d9e346f741d0c12a4b2119920dadae56814bbe793daedf not found: ID does not exist"
Oct 31 00:58:29.654355 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.654332    4935 scope.go:122] "RemoveContainer" containerID="913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c"
Oct 31 00:58:29.661677 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.661663    4935 scope.go:122] "RemoveContainer" containerID="f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f"
Oct 31 00:58:29.669438 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.669422    4935 scope.go:122] "RemoveContainer" containerID="913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c"
Oct 31 00:58:29.669658 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:58:29.669640    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c\": container with ID starting with 913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c not found: ID does not exist" containerID="913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c"
Oct 31 00:58:29.669733 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.669668    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c"} err="failed to get container status \"913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c\": rpc error: code = NotFound desc = could not find container \"913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c\": container with ID starting with 913b8a189dc88791503df5e1df5b5ab670a8b170435fb1f1cbb50ddd4148ea6c not found: ID does not exist"
Oct 31 00:58:29.669733 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.669690    4935 scope.go:122] "RemoveContainer" containerID="f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f"
Oct 31 00:58:29.669926 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:58:29.669909    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f\": container with ID starting with f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f not found: ID does not exist" containerID="f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f"
Oct 31 00:58:29.669966 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.669934    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f"} err="failed to get container status \"f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f\": rpc error: code = NotFound desc = could not find container \"f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f\": container with ID starting with f0f7f45d7dca8a46b8f7709b140cb12c0538149a585ce9cb68a7f70cf4c2883f not found: ID does not exist"
Oct 31 00:58:29.839584 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.839514    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"]
Oct 31 00:58:29.886291 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.886264    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-h2nt2"]
Oct 31 00:58:29.942641 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.942614    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" path="/var/lib/kubelet/pods/3f48454a-47d5-4f0b-b215-c64be8f3baa4/volumes"
Oct 31 00:58:29.951364 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.951345    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"]
Oct 31 00:58:29.957609 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:29.957588    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kq5fz"]
Oct 31 00:58:31.943388 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:31.943356    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" path="/var/lib/kubelet/pods/1d45ed09-bb56-4f2e-aa5e-1aceab5ce105/volumes"
Oct 31 00:58:44.152193 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:58:44.152149    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:58:48.108971 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:48.108921    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:58:48.120489 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:58:48.120467    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 00:59:08.927540 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:08.927508    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"]
Oct 31 00:59:08.928051 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:08.927755    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy" containerID="cri-o://27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d" gracePeriod=30
Oct 31 00:59:14.154186 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:59:14.154147    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 00:59:14.186074 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.186051    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:59:14.293278 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.293248    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-podinfo\") pod \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") "
Oct 31 00:59:14.293419 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.293292    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-envoy\") pod \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") "
Oct 31 00:59:14.293419 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.293335    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-socket\") pod \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") "
Oct 31 00:59:14.293419 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.293379    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-token\" (UniqueName: \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-token\") pod \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") "
Oct 31 00:59:14.293582 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.293417    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istiod-ca-cert\") pod \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") "
Oct 31 00:59:14.293582 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.293464    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-data\") pod \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") "
Oct 31 00:59:14.293582 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.293497    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-certs\") pod \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") "
Oct 31 00:59:14.293582 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.293545    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-credential-socket\") pod \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") "
Oct 31 00:59:14.293764 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.293578    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-kube-api-access-q45zf\" (UniqueName: \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-kube-api-access-q45zf\") pod \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\" (UID: \"441a289f-eb6a-4af9-93e7-30ededcbf8f1\") "
Oct 31 00:59:14.294245 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.294081    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-certs" pod "441a289f-eb6a-4af9-93e7-30ededcbf8f1" (UID: "441a289f-eb6a-4af9-93e7-30ededcbf8f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:14.294245 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.294098    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-data" pod "441a289f-eb6a-4af9-93e7-30ededcbf8f1" (UID: "441a289f-eb6a-4af9-93e7-30ededcbf8f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:14.294622 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.294446    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istiod-ca-cert" pod "441a289f-eb6a-4af9-93e7-30ededcbf8f1" (UID: "441a289f-eb6a-4af9-93e7-30ededcbf8f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:59:14.294622 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.294508    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-socket" pod "441a289f-eb6a-4af9-93e7-30ededcbf8f1" (UID: "441a289f-eb6a-4af9-93e7-30ededcbf8f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:14.294622 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.294557    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-credential-socket" pod "441a289f-eb6a-4af9-93e7-30ededcbf8f1" (UID: "441a289f-eb6a-4af9-93e7-30ededcbf8f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:14.295737 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.295706    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-podinfo" pod "441a289f-eb6a-4af9-93e7-30ededcbf8f1" (UID: "441a289f-eb6a-4af9-93e7-30ededcbf8f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:59:14.296025 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.295997    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-kube-api-access-q45zf" pod "441a289f-eb6a-4af9-93e7-30ededcbf8f1" (UID: "441a289f-eb6a-4af9-93e7-30ededcbf8f1"). InnerVolumeSpecName "kube-api-access-q45zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:59:14.296150 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.296126    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-envoy" pod "441a289f-eb6a-4af9-93e7-30ededcbf8f1" (UID: "441a289f-eb6a-4af9-93e7-30ededcbf8f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:59:14.296150 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.296131    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-token" pod "441a289f-eb6a-4af9-93e7-30ededcbf8f1" (UID: "441a289f-eb6a-4af9-93e7-30ededcbf8f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:59:14.394980 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.394951    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:59:14.394980 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.394974    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:59:14.394980 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.394984    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:59:14.395232 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.394994    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:59:14.395232 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.395003    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q45zf\" (UniqueName: \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-kube-api-access-q45zf\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:59:14.395232 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.395012    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:59:14.395232 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.395019    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:59:14.395232 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.395029    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441a289f-eb6a-4af9-93e7-30ededcbf8f1-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:59:14.395232 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.395053    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441a289f-eb6a-4af9-93e7-30ededcbf8f1-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:59:14.837786 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.837756    4935 generic.go:356] "Generic (PLEG): container finished" podID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerID="27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d" exitCode=0
Oct 31 00:59:14.837945 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.837818    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"
Oct 31 00:59:14.837945 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.837835    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" event={"ID":"441a289f-eb6a-4af9-93e7-30ededcbf8f1","Type":"ContainerDied","Data":"27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d"}
Oct 31 00:59:14.837945 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.837887    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw" event={"ID":"441a289f-eb6a-4af9-93e7-30ededcbf8f1","Type":"ContainerDied","Data":"97c6c56c4ddcc8f0ff1608d7a78a879df4a318ffb2d0eb9a54f49c2342c2ab4d"}
Oct 31 00:59:14.837945 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.837909    4935 scope.go:122] "RemoveContainer" containerID="27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d"
Oct 31 00:59:14.847310 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.847291    4935 scope.go:122] "RemoveContainer" containerID="ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c"
Oct 31 00:59:14.855133 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.855112    4935 scope.go:122] "RemoveContainer" containerID="27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d"
Oct 31 00:59:14.855386 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:59:14.855368    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d\": container with ID starting with 27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d not found: ID does not exist" containerID="27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d"
Oct 31 00:59:14.855445 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.855394    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d"} err="failed to get container status \"27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d\": rpc error: code = NotFound desc = could not find container \"27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d\": container with ID starting with 27fa089f9f7ac104d2bcbdb4ac4f2a9a0345e8626a95973d4461fc1d2d2ae50d not found: ID does not exist"
Oct 31 00:59:14.855445 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.855411    4935 scope.go:122] "RemoveContainer" containerID="ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c"
Oct 31 00:59:14.855656 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:59:14.855640    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c\": container with ID starting with ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c not found: ID does not exist" containerID="ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c"
Oct 31 00:59:14.855700 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.855662    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c"} err="failed to get container status \"ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c\": rpc error: code = NotFound desc = could not find container \"ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c\": container with ID starting with ae0feb4c7e0464d196f332a417369e04772247f0618b07ce7dac279cb9aa6b2c not found: ID does not exist"
Oct 31 00:59:14.876556 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:59:14.876532    4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441a289f_eb6a_4af9_93e7_30ededcbf8f1.slice/crio-97c6c56c4ddcc8f0ff1608d7a78a879df4a318ffb2d0eb9a54f49c2342c2ab4d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441a289f_eb6a_4af9_93e7_30ededcbf8f1.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 00:59:14.996412 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:14.996390    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"]
Oct 31 00:59:15.043147 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:15.043123    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-j4vnw"]
Oct 31 00:59:15.942181 ip-10-0-132-125 kubenswrapper[4935]: I1031 00:59:15.942151    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" path="/var/lib/kubelet/pods/441a289f-eb6a-4af9-93e7-30ededcbf8f1/volumes"
Oct 31 00:59:44.150949 ip-10-0-132-125 kubenswrapper[4935]: E1031 00:59:44.150912    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:00:14.151320 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:00:14.151280    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:00:15.327024 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.326991    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"]
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327340    4935 cpu_manager.go:414] "containerMap: removing container" podUID="68feaa40-b98f-464a-84e6-db602a88ba66" containerName="storage-initializer"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327352    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68feaa40-b98f-464a-84e6-db602a88ba66" containerName="storage-initializer"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327365    4935 cpu_manager.go:414] "containerMap: removing container" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327370    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327376    4935 cpu_manager.go:414] "containerMap: removing container" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327381    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327394    4935 cpu_manager.go:414] "containerMap: removing container" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327398    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327404    4935 cpu_manager.go:414] "containerMap: removing container" podUID="74370c27-079f-4db1-9fe9-9c3195f34fbd" containerName="storage-initializer"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327410    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74370c27-079f-4db1-9fe9-9c3195f34fbd" containerName="storage-initializer"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327420    4935 cpu_manager.go:414] "containerMap: removing container" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327424    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327431    4935 cpu_manager.go:414] "containerMap: removing container" podUID="74370c27-079f-4db1-9fe9-9c3195f34fbd" containerName="main"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327435    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74370c27-079f-4db1-9fe9-9c3195f34fbd" containerName="main"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327442    4935 cpu_manager.go:414] "containerMap: removing container" podUID="68feaa40-b98f-464a-84e6-db602a88ba66" containerName="main"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327446    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68feaa40-b98f-464a-84e6-db602a88ba66" containerName="main"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327458    4935 cpu_manager.go:414] "containerMap: removing container" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="storage-initializer"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327462    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="storage-initializer"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327472    4935 cpu_manager.go:414] "containerMap: removing container" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327478    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327484    4935 cpu_manager.go:414] "containerMap: removing container" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327488    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327537    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327542    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327548    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="981d44c9-81ea-41be-98d5-0a21f14916a2" containerName="main"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327555    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327561    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327568    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d45ed09-bb56-4f2e-aa5e-1aceab5ce105" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327575    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="441a289f-eb6a-4af9-93e7-30ededcbf8f1" containerName="istio-proxy"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327579    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="74370c27-079f-4db1-9fe9-9c3195f34fbd" containerName="main"
Oct 31 01:00:15.329426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.327583    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="68feaa40-b98f-464a-84e6-db602a88ba66" containerName="main"
Oct 31 01:00:15.330589 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.330571    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.332989 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.332969    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.333156 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.332996    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.333156 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.333017    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.333273 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.333157    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/443a018e-d551-48e8-b263-f689ee7ac147-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.333273 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.333203    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqr8k\" (UniqueName: \"kubernetes.io/projected/443a018e-d551-48e8-b263-f689ee7ac147-kube-api-access-rqr8k\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.333371 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.333286    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.427724 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.427684    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-87647\""
Oct 31 01:00:15.427871 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.427751    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Oct 31 01:00:15.427871 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.427771    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Oct 31 01:00:15.428117 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.428094    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Oct 31 01:00:15.429600 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.429580    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"]
Oct 31 01:00:15.434213 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.434196    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.434305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.434228    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.434305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.434251    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.434305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.434272    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.434447 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.434313    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/443a018e-d551-48e8-b263-f689ee7ac147-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.434447 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.434334    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqr8k\" (UniqueName: \"kubernetes.io/projected/443a018e-d551-48e8-b263-f689ee7ac147-kube-api-access-rqr8k\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.434601 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.434582    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.434716 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.434697    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.434776 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.434705    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.436487 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.436470    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.436768 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.436749    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/443a018e-d551-48e8-b263-f689ee7ac147-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.494622 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.494599    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqr8k\" (UniqueName: \"kubernetes.io/projected/443a018e-d551-48e8-b263-f689ee7ac147-kube-api-access-rqr8k\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:15.643335 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:15.643281    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:16.014393 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:16.014364    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"]
Oct 31 01:00:16.021175 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:00:16.021148    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443a018e_d551_48e8_b263_f689ee7ac147.slice/crio-71fe1eb86490a1a1b42261529331480604fb15c83f0cee0c167f7567995a2473 WatchSource:0}: Error finding container 71fe1eb86490a1a1b42261529331480604fb15c83f0cee0c167f7567995a2473: Status 404 returned error can't find the container with id 71fe1eb86490a1a1b42261529331480604fb15c83f0cee0c167f7567995a2473
Oct 31 01:00:16.096114 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:16.096088    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn" event={"ID":"443a018e-d551-48e8-b263-f689ee7ac147","Type":"ContainerStarted","Data":"a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6"}
Oct 31 01:00:16.096234 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:16.096140    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn" event={"ID":"443a018e-d551-48e8-b263-f689ee7ac147","Type":"ContainerStarted","Data":"71fe1eb86490a1a1b42261529331480604fb15c83f0cee0c167f7567995a2473"}
Oct 31 01:00:16.209307 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:16.209261    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn" podStartSLOduration=1.209248179 podStartE2EDuration="1.209248179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:16.208358828 +0000 UTC m=+1888.390888104" watchObservedRunningTime="2026-10-31 01:00:16.209248179 +0000 UTC m=+1888.391777472"
Oct 31 01:00:16.643993 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:16.643967    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:25.644354 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:25.644322    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:25.656421 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:25.656394    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:26.150124 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:26.150094    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:44.152587 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:00:44.152549    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:00:55.483853 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.483819    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"]
Oct 31 01:00:55.493728 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:00:55.493680    4935 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6" cmd=["/bin/sleep","15"]
Oct 31 01:00:55.493858 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.493754    4935 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 01:00:55.493858 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:00:55.493771    4935 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn" podUID="443a018e-d551-48e8-b263-f689ee7ac147" containerName="main" containerID="cri-o://a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6"
Oct 31 01:00:55.493858 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.493827    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn" podUID="443a018e-d551-48e8-b263-f689ee7ac147" containerName="main" containerID="cri-o://a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6" gracePeriod=60
Oct 31 01:00:55.854585 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.854562    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:55.936917 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.936897    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443a018e-d551-48e8-b263-f689ee7ac147-kube-api-access-rqr8k\" (UniqueName: \"kubernetes.io/projected/443a018e-d551-48e8-b263-f689ee7ac147-kube-api-access-rqr8k\") pod \"443a018e-d551-48e8-b263-f689ee7ac147\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") "
Oct 31 01:00:55.937056 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.936938    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-dshm\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-dshm\") pod \"443a018e-d551-48e8-b263-f689ee7ac147\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") "
Oct 31 01:00:55.937056 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.936979    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-home\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-home\") pod \"443a018e-d551-48e8-b263-f689ee7ac147\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") "
Oct 31 01:00:55.937147 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.937085    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/443a018e-d551-48e8-b263-f689ee7ac147-tls-certs\" (UniqueName: \"kubernetes.io/secret/443a018e-d551-48e8-b263-f689ee7ac147-tls-certs\") pod \"443a018e-d551-48e8-b263-f689ee7ac147\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") "
Oct 31 01:00:55.937206 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.937164    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-tmp-dir\") pod \"443a018e-d551-48e8-b263-f689ee7ac147\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") "
Oct 31 01:00:55.937261 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.937210    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-model-cache\") pod \"443a018e-d551-48e8-b263-f689ee7ac147\" (UID: \"443a018e-d551-48e8-b263-f689ee7ac147\") "
Oct 31 01:00:55.937261 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.937239    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-home" pod "443a018e-d551-48e8-b263-f689ee7ac147" (UID: "443a018e-d551-48e8-b263-f689ee7ac147"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:55.937431 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.937403    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-tmp-dir" pod "443a018e-d551-48e8-b263-f689ee7ac147" (UID: "443a018e-d551-48e8-b263-f689ee7ac147"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:55.937507 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.937444    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:55.937507 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.937494    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-model-cache" pod "443a018e-d551-48e8-b263-f689ee7ac147" (UID: "443a018e-d551-48e8-b263-f689ee7ac147"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:55.939465 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.939430    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-dshm" pod "443a018e-d551-48e8-b263-f689ee7ac147" (UID: "443a018e-d551-48e8-b263-f689ee7ac147"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:55.939465 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.939447    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443a018e-d551-48e8-b263-f689ee7ac147-kube-api-access-rqr8k" pod "443a018e-d551-48e8-b263-f689ee7ac147" (UID: "443a018e-d551-48e8-b263-f689ee7ac147"). InnerVolumeSpecName "kube-api-access-rqr8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:55.939606 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:55.939500    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/443a018e-d551-48e8-b263-f689ee7ac147-tls-certs" pod "443a018e-d551-48e8-b263-f689ee7ac147" (UID: "443a018e-d551-48e8-b263-f689ee7ac147"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:00:56.038225 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.038168    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqr8k\" (UniqueName: \"kubernetes.io/projected/443a018e-d551-48e8-b263-f689ee7ac147-kube-api-access-rqr8k\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:56.038225 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.038191    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:56.038225 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.038203    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/443a018e-d551-48e8-b263-f689ee7ac147-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:56.038225 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.038211    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:56.038225 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.038219    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/443a018e-d551-48e8-b263-f689ee7ac147-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:56.272328 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.272302    4935 generic.go:356] "Generic (PLEG): container finished" podID="443a018e-d551-48e8-b263-f689ee7ac147" containerID="a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6" exitCode=0
Oct 31 01:00:56.272469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.272341    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn" event={"ID":"443a018e-d551-48e8-b263-f689ee7ac147","Type":"ContainerDied","Data":"a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6"}
Oct 31 01:00:56.272469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.272346    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"
Oct 31 01:00:56.272469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.272390    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn" event={"ID":"443a018e-d551-48e8-b263-f689ee7ac147","Type":"ContainerDied","Data":"71fe1eb86490a1a1b42261529331480604fb15c83f0cee0c167f7567995a2473"}
Oct 31 01:00:56.272469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.272422    4935 scope.go:122] "RemoveContainer" containerID="a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6"
Oct 31 01:00:56.280902 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.280881    4935 scope.go:122] "RemoveContainer" containerID="a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6"
Oct 31 01:00:56.281157 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:00:56.281140    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6\": container with ID starting with a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6 not found: ID does not exist" containerID="a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6"
Oct 31 01:00:56.281210 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.281164    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6"} err="failed to get container status \"a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6\": rpc error: code = NotFound desc = could not find container \"a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6\": container with ID starting with a6f3430b99136fb645a4c942f91680dd109bdd63787f5404231baec9924edda6 not found: ID does not exist"
Oct 31 01:00:56.429780 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.429721    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"]
Oct 31 01:00:56.435876 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:56.435848    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bsqlgn"]
Oct 31 01:00:57.943166 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:00:57.943092    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443a018e-d551-48e8-b263-f689ee7ac147" path="/var/lib/kubelet/pods/443a018e-d551-48e8-b263-f689ee7ac147/volumes"
Oct 31 01:01:08.234798 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.234768    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"]
Oct 31 01:01:08.235194 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.235104    4935 cpu_manager.go:414] "containerMap: removing container" podUID="443a018e-d551-48e8-b263-f689ee7ac147" containerName="main"
Oct 31 01:01:08.235194 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.235116    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443a018e-d551-48e8-b263-f689ee7ac147" containerName="main"
Oct 31 01:01:08.235194 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.235141    4935 cpu_manager.go:414] "containerMap: removing container" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy"
Oct 31 01:01:08.235194 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.235145    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f48454a-47d5-4f0b-b215-c64be8f3baa4" containerName="istio-proxy"
Oct 31 01:01:08.235322 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.235203    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="443a018e-d551-48e8-b263-f689ee7ac147" containerName="main"
Oct 31 01:01:08.239278 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.239259    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.322233 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.322205    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:01:08.322392 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.322369    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Oct 31 01:01:08.322392 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.322386    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-m9l4n\""
Oct 31 01:01:08.322480 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.322437    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Oct 31 01:01:08.334849 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.334826    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrsp\" (UniqueName: \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-kube-api-access-msrsp\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.334967 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.334872    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.334967 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.334893    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-certs\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.335108 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.334984    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-data\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.335176 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.335150    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-credential-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.335225 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.335196    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-envoy\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.335307 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.335284    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-token\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.335355 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.335338    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec523c08-55f7-4484-9227-8ad7375a3ab8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.335395 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.335384    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-podinfo\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.338287 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.338266    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"]
Oct 31 01:01:08.436723 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.436701    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.436841 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.436728    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-certs\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.436841 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.436745    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-data\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.436841 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.436768    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-credential-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.436841 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.436783    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-envoy\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.436841 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.436805    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-token\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.437135 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.436848    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec523c08-55f7-4484-9227-8ad7375a3ab8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.437135 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.436889    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-podinfo\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.437237 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.437166    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrsp\" (UniqueName: \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-kube-api-access-msrsp\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.437237 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.437223    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.437347 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.437250    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-credential-socket\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.437450 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.437412    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-data\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.437450 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.437435    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-certs\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.437821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.437802    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec523c08-55f7-4484-9227-8ad7375a3ab8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.439338 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.439318    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-envoy\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.439461 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.439445    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-podinfo\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.497028 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.497004    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-token\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.497028 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.497024    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrsp\" (UniqueName: \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-kube-api-access-msrsp\") pod \"router-gateway-1-openshift-default-746559c7f6-bwlr8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.551009 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.550984    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:08.977679 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.977614    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"]
Oct 31 01:01:08.984236 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:01:08.984212    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec523c08_55f7_4484_9227_8ad7375a3ab8.slice/crio-5b7982e31c0a66bab5e17f87d6604bdbb7748e4920d8090baedb75b5863c3c9b WatchSource:0}: Error finding container 5b7982e31c0a66bab5e17f87d6604bdbb7748e4920d8090baedb75b5863c3c9b: Status 404 returned error can't find the container with id 5b7982e31c0a66bab5e17f87d6604bdbb7748e4920d8090baedb75b5863c3c9b
Oct 31 01:01:08.986131 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.986101    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:01:08.986223 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.986164    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:01:08.986223 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:08.986191    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:01:09.329868 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:09.329831    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" event={"ID":"ec523c08-55f7-4484-9227-8ad7375a3ab8","Type":"ContainerStarted","Data":"df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7"}
Oct 31 01:01:09.330403 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:09.329896    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" event={"ID":"ec523c08-55f7-4484-9227-8ad7375a3ab8","Type":"ContainerStarted","Data":"5b7982e31c0a66bab5e17f87d6604bdbb7748e4920d8090baedb75b5863c3c9b"}
Oct 31 01:01:09.446705 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:09.446656    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" podStartSLOduration=1.446641227 podStartE2EDuration="1.446641227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:09.445204873 +0000 UTC m=+1941.627734167" watchObservedRunningTime="2026-10-31 01:01:09.446641227 +0000 UTC m=+1941.629170521"
Oct 31 01:01:09.551619 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:09.551593    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:10.556361 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:10.556333    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:11.339184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:11.339157    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:11.339882 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:11.339861    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:14.151163 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:01:14.151126    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:01:27.382194 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:27.382152    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:01:29.416295 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:29.416200    4935 generic.go:356] "Generic (PLEG): container finished" podID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerID="df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7" exitCode=0
Oct 31 01:01:29.416295 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:29.416238    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" event={"ID":"ec523c08-55f7-4484-9227-8ad7375a3ab8","Type":"ContainerDied","Data":"df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7"}
Oct 31 01:01:29.416715 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:29.416559    4935 scope.go:122] "RemoveContainer" containerID="df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7"
Oct 31 01:01:29.417607 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:29.417577    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:01:29.417666 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:29.417647    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:01:29.417700 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:29.417676    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:01:30.422437 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:30.422405    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" event={"ID":"ec523c08-55f7-4484-9227-8ad7375a3ab8","Type":"ContainerStarted","Data":"a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b"}
Oct 31 01:01:30.563061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:30.556394    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:30.563061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:30.556511    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 01:01:31.566872 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:31.566449    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 01:01:32.571534 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:32.571490    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 01:01:33.564062 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:33.561180    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 01:01:34.561489 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:34.561444    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 01:01:35.555295 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:35.555266    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:35.555570 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:35.555548    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:35.556261 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:01:35.556242    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:01:44.152366 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:01:44.152330    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:02:14.150750 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:02:14.150711    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:02:42.022318 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:42.022275    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"]
Oct 31 01:02:42.022856 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:42.022533    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy" containerID="cri-o://a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b" gracePeriod=30
Oct 31 01:02:44.151553 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:02:44.151514    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:02:44.568289 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.568259    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"]
Oct 31 01:02:44.576548 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.576517    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.624148 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.624121    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"]
Oct 31 01:02:44.669207 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.669180    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Oct 31 01:02:44.669207 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.669178    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Oct 31 01:02:44.669362 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.669183    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-m5prz\""
Oct 31 01:02:44.669362 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.669235    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Oct 31 01:02:44.669362 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.669188    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Oct 31 01:02:44.669362 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.669184    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:02:44.671589 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.671567    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617a72b0-1c64-4dea-a272-8df2551d323c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.671691 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.671615    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.671751 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.671697    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.671793 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.671768    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.671833 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.671789    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.671833 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.671806    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwz7v\" (UniqueName: \"kubernetes.io/projected/617a72b0-1c64-4dea-a272-8df2551d323c-kube-api-access-mwz7v\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.671833 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.671821    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617a72b0-1c64-4dea-a272-8df2551d323c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.671963 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.671914    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773186 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773161    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773312 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773202    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773312 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773219    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773312 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773236    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwz7v\" (UniqueName: \"kubernetes.io/projected/617a72b0-1c64-4dea-a272-8df2551d323c-kube-api-access-mwz7v\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773312 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773251    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617a72b0-1c64-4dea-a272-8df2551d323c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773312 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773287    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773578 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773320    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617a72b0-1c64-4dea-a272-8df2551d323c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773578 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773358    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773678 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773606    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773741 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773678    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773794 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773773    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.773852 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773814    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.774073 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.773989    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617a72b0-1c64-4dea-a272-8df2551d323c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.775703 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.775675    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.775926 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.775908    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617a72b0-1c64-4dea-a272-8df2551d323c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.833525 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.833474    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwz7v\" (UniqueName: \"kubernetes.io/projected/617a72b0-1c64-4dea-a272-8df2551d323c-kube-api-access-mwz7v\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:44.891403 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:44.891378    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:45.375508 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:45.375467    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"]
Oct 31 01:02:45.381955 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:02:45.381928    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617a72b0_1c64_4dea_a272_8df2551d323c.slice/crio-e51e9ea711b37b78a15c1627a0eb77240279e8fb24ecf2d970f5b26f73f9be4a WatchSource:0}: Error finding container e51e9ea711b37b78a15c1627a0eb77240279e8fb24ecf2d970f5b26f73f9be4a: Status 404 returned error can't find the container with id e51e9ea711b37b78a15c1627a0eb77240279e8fb24ecf2d970f5b26f73f9be4a
Oct 31 01:02:45.757050 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:45.757005    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" event={"ID":"617a72b0-1c64-4dea-a272-8df2551d323c","Type":"ContainerStarted","Data":"cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc"}
Oct 31 01:02:45.757210 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:45.757083    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" event={"ID":"617a72b0-1c64-4dea-a272-8df2551d323c","Type":"ContainerStarted","Data":"e51e9ea711b37b78a15c1627a0eb77240279e8fb24ecf2d970f5b26f73f9be4a"}
Oct 31 01:02:47.295540 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.295475    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:02:47.399690 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.399652    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-data\") pod \"ec523c08-55f7-4484-9227-8ad7375a3ab8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") "
Oct 31 01:02:47.399871 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.399745    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-podinfo\") pod \"ec523c08-55f7-4484-9227-8ad7375a3ab8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") "
Oct 31 01:02:47.399871 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.399784    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec523c08-55f7-4484-9227-8ad7375a3ab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec523c08-55f7-4484-9227-8ad7375a3ab8-istiod-ca-cert\") pod \"ec523c08-55f7-4484-9227-8ad7375a3ab8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") "
Oct 31 01:02:47.399871 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.399832    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-socket\") pod \"ec523c08-55f7-4484-9227-8ad7375a3ab8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") "
Oct 31 01:02:47.400081 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.399900    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-kube-api-access-msrsp\" (UniqueName: \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-kube-api-access-msrsp\") pod \"ec523c08-55f7-4484-9227-8ad7375a3ab8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") "
Oct 31 01:02:47.400081 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.399919    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-data" pod "ec523c08-55f7-4484-9227-8ad7375a3ab8" (UID: "ec523c08-55f7-4484-9227-8ad7375a3ab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:47.400081 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.399944    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-envoy\") pod \"ec523c08-55f7-4484-9227-8ad7375a3ab8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") "
Oct 31 01:02:47.400081 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.399981    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-token\" (UniqueName: \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-token\") pod \"ec523c08-55f7-4484-9227-8ad7375a3ab8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") "
Oct 31 01:02:47.400081 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.400062    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-certs\") pod \"ec523c08-55f7-4484-9227-8ad7375a3ab8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") "
Oct 31 01:02:47.400532 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.400096    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-credential-socket\") pod \"ec523c08-55f7-4484-9227-8ad7375a3ab8\" (UID: \"ec523c08-55f7-4484-9227-8ad7375a3ab8\") "
Oct 31 01:02:47.400532 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.400168    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-socket" pod "ec523c08-55f7-4484-9227-8ad7375a3ab8" (UID: "ec523c08-55f7-4484-9227-8ad7375a3ab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:47.400532 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.400344    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-certs" pod "ec523c08-55f7-4484-9227-8ad7375a3ab8" (UID: "ec523c08-55f7-4484-9227-8ad7375a3ab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:47.400532 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.400411    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:47.400532 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.400432    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:47.400532 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.400449    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:47.400532 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.400479    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-credential-socket" pod "ec523c08-55f7-4484-9227-8ad7375a3ab8" (UID: "ec523c08-55f7-4484-9227-8ad7375a3ab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:47.401025 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.400746    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec523c08-55f7-4484-9227-8ad7375a3ab8-istiod-ca-cert" pod "ec523c08-55f7-4484-9227-8ad7375a3ab8" (UID: "ec523c08-55f7-4484-9227-8ad7375a3ab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:02:47.402336 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.402311    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-podinfo" pod "ec523c08-55f7-4484-9227-8ad7375a3ab8" (UID: "ec523c08-55f7-4484-9227-8ad7375a3ab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:02:47.402454 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.402352    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-envoy" pod "ec523c08-55f7-4484-9227-8ad7375a3ab8" (UID: "ec523c08-55f7-4484-9227-8ad7375a3ab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:47.402454 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.402371    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-kube-api-access-msrsp" pod "ec523c08-55f7-4484-9227-8ad7375a3ab8" (UID: "ec523c08-55f7-4484-9227-8ad7375a3ab8"). InnerVolumeSpecName "kube-api-access-msrsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:02:47.402690 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.402673    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-token" pod "ec523c08-55f7-4484-9227-8ad7375a3ab8" (UID: "ec523c08-55f7-4484-9227-8ad7375a3ab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:02:47.501810 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.501784    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:47.501810 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.501811    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:47.501954 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.501820    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec523c08-55f7-4484-9227-8ad7375a3ab8-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:47.501954 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.501831    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msrsp\" (UniqueName: \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-kube-api-access-msrsp\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:47.501954 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.501839    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:47.501954 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.501849    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec523c08-55f7-4484-9227-8ad7375a3ab8-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:47.769727 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.769694    4935 generic.go:356] "Generic (PLEG): container finished" podID="617a72b0-1c64-4dea-a272-8df2551d323c" containerID="cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc" exitCode=0
Oct 31 01:02:47.769851 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.769760    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" event={"ID":"617a72b0-1c64-4dea-a272-8df2551d323c","Type":"ContainerDied","Data":"cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc"}
Oct 31 01:02:47.771440 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.771412    4935 generic.go:356] "Generic (PLEG): container finished" podID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerID="a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b" exitCode=0
Oct 31 01:02:47.771666 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.771467    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"
Oct 31 01:02:47.771666 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.771487    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" event={"ID":"ec523c08-55f7-4484-9227-8ad7375a3ab8","Type":"ContainerDied","Data":"a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b"}
Oct 31 01:02:47.771666 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.771537    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8" event={"ID":"ec523c08-55f7-4484-9227-8ad7375a3ab8","Type":"ContainerDied","Data":"5b7982e31c0a66bab5e17f87d6604bdbb7748e4920d8090baedb75b5863c3c9b"}
Oct 31 01:02:47.771666 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.771560    4935 scope.go:122] "RemoveContainer" containerID="a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b"
Oct 31 01:02:47.780738 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.780723    4935 scope.go:122] "RemoveContainer" containerID="df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7"
Oct 31 01:02:47.788325 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.788310    4935 scope.go:122] "RemoveContainer" containerID="a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b"
Oct 31 01:02:47.788623 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:02:47.788602    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b\": container with ID starting with a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b not found: ID does not exist" containerID="a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b"
Oct 31 01:02:47.788710 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.788634    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b"} err="failed to get container status \"a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b\": rpc error: code = NotFound desc = could not find container \"a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b\": container with ID starting with a1872db77d4a0e1382a6cc355ca448f150860644ba0539aa87555697e116240b not found: ID does not exist"
Oct 31 01:02:47.788710 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.788657    4935 scope.go:122] "RemoveContainer" containerID="df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7"
Oct 31 01:02:47.788918 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:02:47.788901    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7\": container with ID starting with df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7 not found: ID does not exist" containerID="df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7"
Oct 31 01:02:47.788983 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.788929    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7"} err="failed to get container status \"df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7\": rpc error: code = NotFound desc = could not find container \"df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7\": container with ID starting with df06de3ef88396d75bbf6ce0681affe96f110114626d0a72db813b8d33c364b7 not found: ID does not exist"
Oct 31 01:02:47.996598 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:47.996564    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"]
Oct 31 01:02:48.002616 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:48.002586    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-746559c7f6-bwlr8"]
Oct 31 01:02:48.777581 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:48.777545    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" event={"ID":"617a72b0-1c64-4dea-a272-8df2551d323c","Type":"ContainerStarted","Data":"ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3"}
Oct 31 01:02:48.892205 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:48.892171    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:48.989593 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:48.989545    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podStartSLOduration=2.940032064 podStartE2EDuration="4.989528157s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.049496093s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:48.941807957 +0000 UTC m=+2041.124337252" watchObservedRunningTime="2026-10-31 01:02:48.989528157 +0000 UTC m=+2041.172057451"
Oct 31 01:02:49.943326 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:49.943287    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" path="/var/lib/kubelet/pods/ec523c08-55f7-4484-9227-8ad7375a3ab8/volumes"
Oct 31 01:02:54.891699 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:54.891665    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:02:54.893307 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:02:54.893274    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 01:03:04.891854 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:04.891801    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 01:03:10.859050 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.859002    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"]
Oct 31 01:03:10.859425 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.859361    4935 cpu_manager.go:414] "containerMap: removing container" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy"
Oct 31 01:03:10.859425 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.859374    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy"
Oct 31 01:03:10.859425 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.859390    4935 cpu_manager.go:414] "containerMap: removing container" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy"
Oct 31 01:03:10.859425 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.859397    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy"
Oct 31 01:03:10.859555 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.859470    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy"
Oct 31 01:03:10.859555 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.859479    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec523c08-55f7-4484-9227-8ad7375a3ab8" containerName="istio-proxy"
Oct 31 01:03:10.862450 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.862434    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:10.905775 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.905747    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"]
Oct 31 01:03:10.909114 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.909098    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:10.925267 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.925248    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:03:10.925267 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.925258    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-q9j8d\""
Oct 31 01:03:10.960699 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.960673    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"]
Oct 31 01:03:10.968437 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:10.968411    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"]
Oct 31 01:03:11.002926 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.002898    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.003029 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.002928    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4g6l\" (UniqueName: \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-kube-api-access-z4g6l\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.003029 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.002955    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.003029 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003012    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.003162 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003130    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.003195 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003171    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.003227 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003208    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.003260 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003240    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.003405 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003334    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.003447 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003424    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.003498 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003455    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.003549 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003532    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.003597 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003582    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.003698 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003676    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.003754 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003737    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.003804 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003766    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.003871 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003852    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.003912 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.003898    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82w8b\" (UniqueName: \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-kube-api-access-82w8b\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.104470 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104437    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.104661 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104479    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.104661 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104498    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.104661 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104530    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.104661 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104558    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.104661 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104583    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.104661 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104612    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.104661 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104638    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104671    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104719    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104745    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104777    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104811    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82w8b\" (UniqueName: \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-kube-api-access-82w8b\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104840    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104863    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4g6l\" (UniqueName: \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-kube-api-access-z4g6l\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104914    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104941    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104958    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104978    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.104992    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.105064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.105005    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.105704 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.105400    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.105704 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.105430    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.105704 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.105557    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.105704 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.105653    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.105704 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.105689    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.106015 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.105834    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.106015 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.105911    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.107816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.107789    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.107937 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.107849    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.108229 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.108211    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.108348 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.108328    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.164857 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.164791    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.164979 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.164916    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4g6l\" (UniqueName: \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-kube-api-access-z4g6l\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.165241 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.165222    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-s4q4v\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.165417 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.165400    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82w8b\" (UniqueName: \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-kube-api-access-82w8b\") pod \"openshift-ai-inference-openshift-default-568c495856-v6lqx\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.175192 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.175172    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:11.221584 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.221387    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:11.603206 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.603173    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"]
Oct 31 01:03:11.606725 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.606703    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"]
Oct 31 01:03:11.609184 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:03:11.609157    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91cc4b2_3d1b_4ac8_945c_34029d5bf5ff.slice/crio-46800759352b89861d7a9acb097bb90f85ee6b7da95fed13692139ca5e0795ec WatchSource:0}: Error finding container 46800759352b89861d7a9acb097bb90f85ee6b7da95fed13692139ca5e0795ec: Status 404 returned error can't find the container with id 46800759352b89861d7a9acb097bb90f85ee6b7da95fed13692139ca5e0795ec
Oct 31 01:03:11.611308 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.611279    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:11.611374 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.611340    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:11.611374 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.611365    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:11.645162 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:03:11.645136    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe47e516_df31_4e4f_8a14_b8c00990c0f6.slice/crio-b40c23b596cbd77b50b31e2dbf45b814836a224c92f753c07e321af7a1b0867d WatchSource:0}: Error finding container b40c23b596cbd77b50b31e2dbf45b814836a224c92f753c07e321af7a1b0867d: Status 404 returned error can't find the container with id b40c23b596cbd77b50b31e2dbf45b814836a224c92f753c07e321af7a1b0867d
Oct 31 01:03:11.647073 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.647016    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:11.647184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.647105    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:11.647184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.647134    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:11.884116 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.883789    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" event={"ID":"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff","Type":"ContainerStarted","Data":"6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5"}
Oct 31 01:03:11.884116 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.883851    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" event={"ID":"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff","Type":"ContainerStarted","Data":"46800759352b89861d7a9acb097bb90f85ee6b7da95fed13692139ca5e0795ec"}
Oct 31 01:03:11.885927 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.885900    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" event={"ID":"fe47e516-df31-4e4f-8a14-b8c00990c0f6","Type":"ContainerStarted","Data":"f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2"}
Oct 31 01:03:11.886083 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:11.885952    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" event={"ID":"fe47e516-df31-4e4f-8a14-b8c00990c0f6","Type":"ContainerStarted","Data":"b40c23b596cbd77b50b31e2dbf45b814836a224c92f753c07e321af7a1b0867d"}
Oct 31 01:03:12.004574 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:12.004511    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" podStartSLOduration=2.00449088 podStartE2EDuration="2.00449088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:12.000953708 +0000 UTC m=+2064.183483027" watchObservedRunningTime="2026-10-31 01:03:12.00449088 +0000 UTC m=+2064.187020175"
Oct 31 01:03:12.119444 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:12.119384    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" podStartSLOduration=2.119368481 podStartE2EDuration="2.119368481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:12.118209706 +0000 UTC m=+2064.300738992" watchObservedRunningTime="2026-10-31 01:03:12.119368481 +0000 UTC m=+2064.301897775"
Oct 31 01:03:12.180927 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:12.180834    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:12.181290 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:12.181267    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-s4q4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:03:12.181451 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:12.181428    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 01:03:12.222479 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:12.222446    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:12.224209 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:12.224183    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-v6lqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:03:12.224404 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:12.224387    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 01:03:13.176480 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:13.176441    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-s4q4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:03:13.177117 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:13.177085    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 01:03:13.222768 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:13.222736    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-v6lqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:03:13.222948 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:13.222788    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 01:03:14.176384 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:14.176235    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-s4q4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:03:14.176384 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:14.176299    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 01:03:14.176384 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:03:14.176348    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:03:14.222736 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:14.222696    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-v6lqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:03:14.222896 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:14.222759    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 01:03:14.896062 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:14.895476    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 01:03:15.178527 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:15.178434    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-s4q4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:03:15.179120 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:15.179086    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 01:03:15.222498 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:15.222466    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-v6lqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:03:15.222712 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:15.222688    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 01:03:16.181064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:16.179433    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-s4q4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:03:16.181064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:16.179495    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 01:03:16.222652 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:16.222608    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-v6lqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:03:16.222824 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:16.222677    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 01:03:17.178061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:17.177152    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-s4q4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:03:17.178061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:17.177211    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 01:03:17.222656 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:17.222423    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-v6lqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 01:03:17.222656 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:17.222483    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 01:03:18.180747 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:18.180717    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:18.181146 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:18.181126    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:18.181897 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:18.181874    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:03:18.226549 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:18.226522    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:18.226876 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:18.226790    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:18.227368 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:18.227352    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:03:24.892020 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:24.891969    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 01:03:26.001898 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.001854    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"]
Oct 31 01:03:26.006462 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.006439    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.189411 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.189373    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.189564 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.189422    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxvb\" (UniqueName: \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-kube-api-access-vjxvb\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.189564 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.189441    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12df961-8d85-4e2f-8f62-bc585de0b55d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.189639 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.189592    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.189675 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.189655    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.189708 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.189689    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.189753 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.189715    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.189798 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.189785    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.189851 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.189836    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.206328 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.206301    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"]
Oct 31 01:03:26.290469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.290395    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.290469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.290439    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxvb\" (UniqueName: \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-kube-api-access-vjxvb\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.290469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.290455    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12df961-8d85-4e2f-8f62-bc585de0b55d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.290734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.290483    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.290734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.290521    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.290734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.290543    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.290734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.290558    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.290734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.290584    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.290734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.290630    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.291063 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.291021    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.291132 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.291062    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.291132 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.291082    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.291241 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.291224    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.291330 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.291309    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12df961-8d85-4e2f-8f62-bc585de0b55d-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.293112 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.293085    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.293324 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.293308    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.349914 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.349890    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.350249 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.350227    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxvb\" (UniqueName: \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-kube-api-access-vjxvb\") pod \"openshift-ai-inference-openshift-default-568c495856-vdhnm\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:26.619496 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:26.619405    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:27.047297 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:27.047258    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"]
Oct 31 01:03:27.053105 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:03:27.053073    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12df961_8d85_4e2f_8f62_bc585de0b55d.slice/crio-4dbc9146900f6949bd60bade5e6ba6cd67e6af266f04b602a6c638788f5f34ef WatchSource:0}: Error finding container 4dbc9146900f6949bd60bade5e6ba6cd67e6af266f04b602a6c638788f5f34ef: Status 404 returned error can't find the container with id 4dbc9146900f6949bd60bade5e6ba6cd67e6af266f04b602a6c638788f5f34ef
Oct 31 01:03:27.055442 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:27.055404    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:27.055519 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:27.055493    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:27.055583 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:27.055567    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:27.997570 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:27.997536    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm" event={"ID":"f12df961-8d85-4e2f-8f62-bc585de0b55d","Type":"ContainerStarted","Data":"580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a"}
Oct 31 01:03:27.997747 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:27.997592    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm" event={"ID":"f12df961-8d85-4e2f-8f62-bc585de0b55d","Type":"ContainerStarted","Data":"4dbc9146900f6949bd60bade5e6ba6cd67e6af266f04b602a6c638788f5f34ef"}
Oct 31 01:03:28.162062 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:28.161577    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm" podStartSLOduration=3.161557182 podStartE2EDuration="3.161557182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:28.160521629 +0000 UTC m=+2080.343050922" watchObservedRunningTime="2026-10-31 01:03:28.161557182 +0000 UTC m=+2080.344086476"
Oct 31 01:03:28.620613 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:28.620578    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:28.622142 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:28.622118    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-vdhnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 31 01:03:28.622264 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:28.622165    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm" podUID="f12df961-8d85-4e2f-8f62-bc585de0b55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 31 01:03:29.620787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:29.620748    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-vdhnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 31 01:03:29.621239 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:29.620808    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm" podUID="f12df961-8d85-4e2f-8f62-bc585de0b55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 31 01:03:30.620581 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:30.620537    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-vdhnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 31 01:03:30.620755 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:30.620597    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm" podUID="f12df961-8d85-4e2f-8f62-bc585de0b55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 31 01:03:31.619994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:31.619964    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-vdhnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 31 01:03:31.620393 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:31.620019    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm" podUID="f12df961-8d85-4e2f-8f62-bc585de0b55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 31 01:03:32.620451 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:32.620418    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-vdhnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 31 01:03:32.620843 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:32.620472    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm" podUID="f12df961-8d85-4e2f-8f62-bc585de0b55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 31 01:03:33.623736 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:33.623701    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:33.624231 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:33.624068    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:33.624676 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:33.624656    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:03:34.891984 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:34.891931    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 01:03:41.145172 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.145093    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"]
Oct 31 01:03:41.148724 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.148704    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.222113 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.222078    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.222113 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.222115    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.222306 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.222134    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh5j8\" (UniqueName: \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-kube-api-access-qh5j8\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.222306 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.222256    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.222306 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.222275    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.222306 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.222291    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.222456 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.222308    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.222456 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.222330    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.222456 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.222441    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.242631 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.242600    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"]
Oct 31 01:03:41.323435 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.323404    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.323435 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.323438    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.323680 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.323458    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.323680 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.323481    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.323680 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.323605    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.323839 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.323682    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.323839 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.323718    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.323839 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.323747    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.323839 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.323779    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh5j8\" (UniqueName: \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-kube-api-access-qh5j8\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.324023 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.323998    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.324105 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.324028    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.324105 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.324072    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.324181 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.324153    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.324362 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.324339    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.326150 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.326128    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.326432 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.326409    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.383294 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.383267    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.383551 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.383530    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh5j8\" (UniqueName: \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-kube-api-access-qh5j8\") pod \"openshift-ai-inference-openshift-default-568c495856-6g8bg\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.462519 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.462443    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:41.891621 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.891578    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"]
Oct 31 01:03:41.897872 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:03:41.897841    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccee86e_ea66_44ba_8fc7_a81cbe82d094.slice/crio-769fe50dd83787683f9a7b4e11b9938b485657f53d98c0785728b0cb2c6b2801 WatchSource:0}: Error finding container 769fe50dd83787683f9a7b4e11b9938b485657f53d98c0785728b0cb2c6b2801: Status 404 returned error can't find the container with id 769fe50dd83787683f9a7b4e11b9938b485657f53d98c0785728b0cb2c6b2801
Oct 31 01:03:41.900104 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.900068    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:41.900206 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.900135    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:41.900206 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:41.900162    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:03:42.073772 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:42.073738    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" event={"ID":"9ccee86e-ea66-44ba-8fc7-a81cbe82d094","Type":"ContainerStarted","Data":"2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5"}
Oct 31 01:03:42.073935 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:42.073795    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" event={"ID":"9ccee86e-ea66-44ba-8fc7-a81cbe82d094","Type":"ContainerStarted","Data":"769fe50dd83787683f9a7b4e11b9938b485657f53d98c0785728b0cb2c6b2801"}
Oct 31 01:03:42.190882 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:42.190762    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" podStartSLOduration=1.190743149 podStartE2EDuration="1.190743149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:42.18979678 +0000 UTC m=+2094.372326073" watchObservedRunningTime="2026-10-31 01:03:42.190743149 +0000 UTC m=+2094.373272443"
Oct 31 01:03:42.464503 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:42.464410    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:42.470061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:42.468194    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-6g8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:03:42.470061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:42.468266    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 01:03:43.463062 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:43.462997    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-6g8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:03:43.463679 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:43.463071    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 01:03:44.186677 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:03:44.186639    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:03:44.464116 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:44.463489    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-6g8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:03:44.464116 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:44.463552    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 01:03:44.891906 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:44.891864    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 01:03:45.467478 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:45.467439    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-6g8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:03:45.468081 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:45.468029    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 01:03:46.466545 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:46.466502    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-6g8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:03:46.466851 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:46.466819    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 01:03:47.466479 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:47.466438    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-6g8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:03:47.467103 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:47.467070    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 01:03:48.140234 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:48.140206    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:03:48.152542 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:48.152512    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:03:48.466774 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:48.466700    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:48.467169 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:48.467081    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:48.467775 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:48.467756    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:03:54.892261 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:03:54.892217    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 01:04:04.892469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:04.892424    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 01:04:14.154978 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:04:14.154938    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:04:14.892146 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:14.892100    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 01:04:24.891716 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:24.891664    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 31 01:04:34.901238 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:34.901208    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:04:34.908442 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:34.908418    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:04:43.005476 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:43.005443    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"]
Oct 31 01:04:44.150631 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:04:44.150596    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:04:58.028701 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.028643    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main" containerID="cri-o://ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3" gracePeriod=45
Oct 31 01:04:58.784303 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.784246    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:04:58.931442 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.931415    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-kserve-provision-location\") pod \"617a72b0-1c64-4dea-a272-8df2551d323c\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") "
Oct 31 01:04:58.931565 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.931451    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-model-cache\") pod \"617a72b0-1c64-4dea-a272-8df2551d323c\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") "
Oct 31 01:04:58.931565 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.931510    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617a72b0-1c64-4dea-a272-8df2551d323c-kube-api-access-mwz7v\" (UniqueName: \"kubernetes.io/projected/617a72b0-1c64-4dea-a272-8df2551d323c-kube-api-access-mwz7v\") pod \"617a72b0-1c64-4dea-a272-8df2551d323c\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") "
Oct 31 01:04:58.931565 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.931560    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-dshm\") pod \"617a72b0-1c64-4dea-a272-8df2551d323c\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") "
Oct 31 01:04:58.931703 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.931580    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/617a72b0-1c64-4dea-a272-8df2551d323c-tls-certs\" (UniqueName: \"kubernetes.io/secret/617a72b0-1c64-4dea-a272-8df2551d323c-tls-certs\") pod \"617a72b0-1c64-4dea-a272-8df2551d323c\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") "
Oct 31 01:04:58.931703 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.931604    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/617a72b0-1c64-4dea-a272-8df2551d323c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617a72b0-1c64-4dea-a272-8df2551d323c-cabundle-cert\") pod \"617a72b0-1c64-4dea-a272-8df2551d323c\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") "
Oct 31 01:04:58.931703 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.931653    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-home\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-home\") pod \"617a72b0-1c64-4dea-a272-8df2551d323c\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") "
Oct 31 01:04:58.931703 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.931694    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-tmp-dir\") pod \"617a72b0-1c64-4dea-a272-8df2551d323c\" (UID: \"617a72b0-1c64-4dea-a272-8df2551d323c\") "
Oct 31 01:04:58.931903 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.931758    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-model-cache" pod "617a72b0-1c64-4dea-a272-8df2551d323c" (UID: "617a72b0-1c64-4dea-a272-8df2551d323c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:58.932370 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.932029    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:58.932370 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.932317    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617a72b0-1c64-4dea-a272-8df2551d323c-cabundle-cert" pod "617a72b0-1c64-4dea-a272-8df2551d323c" (UID: "617a72b0-1c64-4dea-a272-8df2551d323c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:58.932545 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.932491    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-home" pod "617a72b0-1c64-4dea-a272-8df2551d323c" (UID: "617a72b0-1c64-4dea-a272-8df2551d323c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:58.934413 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.934384    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-dshm" pod "617a72b0-1c64-4dea-a272-8df2551d323c" (UID: "617a72b0-1c64-4dea-a272-8df2551d323c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:58.934527 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.934477    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617a72b0-1c64-4dea-a272-8df2551d323c-kube-api-access-mwz7v" pod "617a72b0-1c64-4dea-a272-8df2551d323c" (UID: "617a72b0-1c64-4dea-a272-8df2551d323c"). InnerVolumeSpecName "kube-api-access-mwz7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:58.934527 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.934477    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617a72b0-1c64-4dea-a272-8df2551d323c-tls-certs" pod "617a72b0-1c64-4dea-a272-8df2551d323c" (UID: "617a72b0-1c64-4dea-a272-8df2551d323c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:04:58.942444 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.942421    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-tmp-dir" pod "617a72b0-1c64-4dea-a272-8df2551d323c" (UID: "617a72b0-1c64-4dea-a272-8df2551d323c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:58.955009 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:58.954987    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-kserve-provision-location" pod "617a72b0-1c64-4dea-a272-8df2551d323c" (UID: "617a72b0-1c64-4dea-a272-8df2551d323c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:59.033111 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.033089    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:59.033111 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.033109    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:59.033437 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.033120    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:59.033437 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.033129    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwz7v\" (UniqueName: \"kubernetes.io/projected/617a72b0-1c64-4dea-a272-8df2551d323c-kube-api-access-mwz7v\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:59.033437 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.033137    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/617a72b0-1c64-4dea-a272-8df2551d323c-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:59.033437 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.033151    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/617a72b0-1c64-4dea-a272-8df2551d323c-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:59.033437 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.033160    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617a72b0-1c64-4dea-a272-8df2551d323c-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:59.461536 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.461509    4935 generic.go:356] "Generic (PLEG): container finished" podID="617a72b0-1c64-4dea-a272-8df2551d323c" containerID="ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3" exitCode=0
Oct 31 01:04:59.461651 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.461554    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"
Oct 31 01:04:59.461651 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.461588    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" event={"ID":"617a72b0-1c64-4dea-a272-8df2551d323c","Type":"ContainerDied","Data":"ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3"}
Oct 31 01:04:59.461651 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.461637    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx" event={"ID":"617a72b0-1c64-4dea-a272-8df2551d323c","Type":"ContainerDied","Data":"e51e9ea711b37b78a15c1627a0eb77240279e8fb24ecf2d970f5b26f73f9be4a"}
Oct 31 01:04:59.461756 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.461659    4935 scope.go:122] "RemoveContainer" containerID="ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3"
Oct 31 01:04:59.473234 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.473219    4935 scope.go:122] "RemoveContainer" containerID="cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc"
Oct 31 01:04:59.480928 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.480914    4935 scope.go:122] "RemoveContainer" containerID="ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3"
Oct 31 01:04:59.481257 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:04:59.481238    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3\": container with ID starting with ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3 not found: ID does not exist" containerID="ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3"
Oct 31 01:04:59.481304 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.481264    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3"} err="failed to get container status \"ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3\": rpc error: code = NotFound desc = could not find container \"ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3\": container with ID starting with ef476d5c7ca3281b36bd654b37670db5a884e7a513139a668e9e4fc990524da3 not found: ID does not exist"
Oct 31 01:04:59.481304 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.481282    4935 scope.go:122] "RemoveContainer" containerID="cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc"
Oct 31 01:04:59.481513 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:04:59.481493    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc\": container with ID starting with cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc not found: ID does not exist" containerID="cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc"
Oct 31 01:04:59.481558 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.481520    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc"} err="failed to get container status \"cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc\": rpc error: code = NotFound desc = could not find container \"cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc\": container with ID starting with cb836716fab3d33700b8f6f0cccd75b914d6214b4cccad4d14af4c0a2a45efbc not found: ID does not exist"
Oct 31 01:04:59.622999 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.622980    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"]
Oct 31 01:04:59.669523 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.669502    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-76d4f76c5f4klsx"]
Oct 31 01:04:59.948876 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:04:59.948837    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" path="/var/lib/kubelet/pods/617a72b0-1c64-4dea-a272-8df2551d323c/volumes"
Oct 31 01:05:14.151180 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:05:14.151145    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:05:44.151168 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:05:44.151133    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:06:02.505247 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.505215    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"]
Oct 31 01:06:02.505771 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.505603    4935 cpu_manager.go:414] "containerMap: removing container" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="storage-initializer"
Oct 31 01:06:02.505771 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.505614    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="storage-initializer"
Oct 31 01:06:02.505771 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.505628    4935 cpu_manager.go:414] "containerMap: removing container" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main"
Oct 31 01:06:02.505771 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.505633    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main"
Oct 31 01:06:02.505771 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.505688    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="617a72b0-1c64-4dea-a272-8df2551d323c" containerName="main"
Oct 31 01:06:02.509990 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.509970    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.559221 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.559196    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"]
Oct 31 01:06:02.603933 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.603907    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Oct 31 01:06:02.603933 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.603906    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Oct 31 01:06:02.604132 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.604114    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:06:02.604187 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.604174    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Oct 31 01:06:02.604187 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.604178    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-6gd2n\""
Oct 31 01:06:02.604261 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.604245    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Oct 31 01:06:02.636793 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.636769    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8245e69a-c401-4364-b7d1-b236cba101d0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.636915 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.636816    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.636915 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.636834    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.637019 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.636909    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8245e69a-c401-4364-b7d1-b236cba101d0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.637094 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.637016    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq8q8\" (UniqueName: \"kubernetes.io/projected/8245e69a-c401-4364-b7d1-b236cba101d0-kube-api-access-kq8q8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.637094 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.637075    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.637171 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.637125    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.637171 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.637152    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738105 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738074    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq8q8\" (UniqueName: \"kubernetes.io/projected/8245e69a-c401-4364-b7d1-b236cba101d0-kube-api-access-kq8q8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738275 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738115    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738275 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738134    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738275 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738157    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738275 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738177    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8245e69a-c401-4364-b7d1-b236cba101d0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738275 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738226    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738275 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738251    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738562 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738278    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8245e69a-c401-4364-b7d1-b236cba101d0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738652 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738627    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738765 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738742    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738765 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738757    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738869 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738821    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.738988 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.738937    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8245e69a-c401-4364-b7d1-b236cba101d0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.740642 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.740620    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.740803 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.740786    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8245e69a-c401-4364-b7d1-b236cba101d0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.799878 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.799824    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq8q8\" (UniqueName: \"kubernetes.io/projected/8245e69a-c401-4364-b7d1-b236cba101d0-kube-api-access-kq8q8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:02.825398 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:02.825372    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:03.252988 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:03.252957    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"]
Oct 31 01:06:03.259007 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:06:03.258975    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8245e69a_c401_4364_b7d1_b236cba101d0.slice/crio-dd93b5f778fe8e6f00365428e4dce7b4fe277b68dea7fd2b369c5b46ff51cbd8 WatchSource:0}: Error finding container dd93b5f778fe8e6f00365428e4dce7b4fe277b68dea7fd2b369c5b46ff51cbd8: Status 404 returned error can't find the container with id dd93b5f778fe8e6f00365428e4dce7b4fe277b68dea7fd2b369c5b46ff51cbd8
Oct 31 01:06:03.744413 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:03.744370    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" event={"ID":"8245e69a-c401-4364-b7d1-b236cba101d0","Type":"ContainerStarted","Data":"25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066"}
Oct 31 01:06:03.744808 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:03.744455    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" event={"ID":"8245e69a-c401-4364-b7d1-b236cba101d0","Type":"ContainerStarted","Data":"dd93b5f778fe8e6f00365428e4dce7b4fe277b68dea7fd2b369c5b46ff51cbd8"}
Oct 31 01:06:06.759217 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:06.759186    4935 generic.go:356] "Generic (PLEG): container finished" podID="8245e69a-c401-4364-b7d1-b236cba101d0" containerID="25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066" exitCode=0
Oct 31 01:06:06.759584 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:06.759257    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" event={"ID":"8245e69a-c401-4364-b7d1-b236cba101d0","Type":"ContainerDied","Data":"25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066"}
Oct 31 01:06:07.767239 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:07.767206    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" event={"ID":"8245e69a-c401-4364-b7d1-b236cba101d0","Type":"ContainerStarted","Data":"fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078"}
Oct 31 01:06:07.826260 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:07.826227    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:07.980698 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:07.980631    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" podStartSLOduration=2.654198531 podStartE2EDuration="5.980617324s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.326418793s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:07.933073517 +0000 UTC m=+2240.115602808" watchObservedRunningTime="2026-10-31 01:06:07.980617324 +0000 UTC m=+2240.163146609"
Oct 31 01:06:12.826493 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:12.826463    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:06:12.827932 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:12.827908    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Oct 31 01:06:14.151971 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:06:14.151928    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:06:22.826149 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:22.826105    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Oct 31 01:06:32.826101 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:32.826021    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Oct 31 01:06:42.826340 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:42.826287    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Oct 31 01:06:44.153918 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:06:44.153880    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:06:52.826029 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:06:52.825981    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Oct 31 01:07:02.826177 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:07:02.826135    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Oct 31 01:07:12.826290 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:07:12.826239    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Oct 31 01:07:14.153289 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:07:14.153243    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:07:22.836337 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:07:22.836307    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:07:22.844603 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:07:22.844576    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:07:44.152618 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:07:44.152571    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:07:52.150980 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:07:52.150941    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"]
Oct 31 01:08:07.174456 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.174414    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="main" containerID="cri-o://fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078" gracePeriod=45
Oct 31 01:08:07.823240 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.823214    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:08:07.885272 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885202    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-tmp-dir\") pod \"8245e69a-c401-4364-b7d1-b236cba101d0\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") "
Oct 31 01:08:07.885272 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885255    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8245e69a-c401-4364-b7d1-b236cba101d0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8245e69a-c401-4364-b7d1-b236cba101d0-cabundle-cert\") pod \"8245e69a-c401-4364-b7d1-b236cba101d0\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") "
Oct 31 01:08:07.885479 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885277    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-model-cache\") pod \"8245e69a-c401-4364-b7d1-b236cba101d0\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") "
Oct 31 01:08:07.885479 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885341    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-kserve-provision-location\") pod \"8245e69a-c401-4364-b7d1-b236cba101d0\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") "
Oct 31 01:08:07.885479 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885377    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-dshm\") pod \"8245e69a-c401-4364-b7d1-b236cba101d0\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") "
Oct 31 01:08:07.885479 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885406    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-home\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-home\") pod \"8245e69a-c401-4364-b7d1-b236cba101d0\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") "
Oct 31 01:08:07.885679 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885499    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8245e69a-c401-4364-b7d1-b236cba101d0-kube-api-access-kq8q8\" (UniqueName: \"kubernetes.io/projected/8245e69a-c401-4364-b7d1-b236cba101d0-kube-api-access-kq8q8\") pod \"8245e69a-c401-4364-b7d1-b236cba101d0\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") "
Oct 31 01:08:07.885679 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885509    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-model-cache" pod "8245e69a-c401-4364-b7d1-b236cba101d0" (UID: "8245e69a-c401-4364-b7d1-b236cba101d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:07.885679 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885529    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8245e69a-c401-4364-b7d1-b236cba101d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/8245e69a-c401-4364-b7d1-b236cba101d0-tls-certs\") pod \"8245e69a-c401-4364-b7d1-b236cba101d0\" (UID: \"8245e69a-c401-4364-b7d1-b236cba101d0\") "
Oct 31 01:08:07.885679 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885636    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8245e69a-c401-4364-b7d1-b236cba101d0-cabundle-cert" pod "8245e69a-c401-4364-b7d1-b236cba101d0" (UID: "8245e69a-c401-4364-b7d1-b236cba101d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:07.885892 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885861    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-home" pod "8245e69a-c401-4364-b7d1-b236cba101d0" (UID: "8245e69a-c401-4364-b7d1-b236cba101d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:07.886007 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.885986    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8245e69a-c401-4364-b7d1-b236cba101d0-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:07.886094 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.886013    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:07.886094 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.886027    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:07.887788 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.887762    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-dshm" pod "8245e69a-c401-4364-b7d1-b236cba101d0" (UID: "8245e69a-c401-4364-b7d1-b236cba101d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:07.888213 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.888187    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8245e69a-c401-4364-b7d1-b236cba101d0-tls-certs" pod "8245e69a-c401-4364-b7d1-b236cba101d0" (UID: "8245e69a-c401-4364-b7d1-b236cba101d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:08:07.888313 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.888209    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8245e69a-c401-4364-b7d1-b236cba101d0-kube-api-access-kq8q8" pod "8245e69a-c401-4364-b7d1-b236cba101d0" (UID: "8245e69a-c401-4364-b7d1-b236cba101d0"). InnerVolumeSpecName "kube-api-access-kq8q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:07.895533 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.895509    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-tmp-dir" pod "8245e69a-c401-4364-b7d1-b236cba101d0" (UID: "8245e69a-c401-4364-b7d1-b236cba101d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:07.909255 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.909229    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-kserve-provision-location" pod "8245e69a-c401-4364-b7d1-b236cba101d0" (UID: "8245e69a-c401-4364-b7d1-b236cba101d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:07.986815 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.986774    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq8q8\" (UniqueName: \"kubernetes.io/projected/8245e69a-c401-4364-b7d1-b236cba101d0-kube-api-access-kq8q8\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:07.986815 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.986813    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8245e69a-c401-4364-b7d1-b236cba101d0-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:07.986976 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.986841    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:07.986976 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.986850    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:07.986976 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:07.986859    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8245e69a-c401-4364-b7d1-b236cba101d0-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:08.350605 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.350571    4935 generic.go:356] "Generic (PLEG): container finished" podID="8245e69a-c401-4364-b7d1-b236cba101d0" containerID="fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078" exitCode=0
Oct 31 01:08:08.351003 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.350609    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" event={"ID":"8245e69a-c401-4364-b7d1-b236cba101d0","Type":"ContainerDied","Data":"fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078"}
Oct 31 01:08:08.351003 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.350617    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"
Oct 31 01:08:08.351003 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.350651    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd" event={"ID":"8245e69a-c401-4364-b7d1-b236cba101d0","Type":"ContainerDied","Data":"dd93b5f778fe8e6f00365428e4dce7b4fe277b68dea7fd2b369c5b46ff51cbd8"}
Oct 31 01:08:08.351003 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.350671    4935 scope.go:122] "RemoveContainer" containerID="fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078"
Oct 31 01:08:08.359792 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.359772    4935 scope.go:122] "RemoveContainer" containerID="25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066"
Oct 31 01:08:08.367773 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.367757    4935 scope.go:122] "RemoveContainer" containerID="fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078"
Oct 31 01:08:08.368054 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:08:08.368017    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078\": container with ID starting with fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078 not found: ID does not exist" containerID="fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078"
Oct 31 01:08:08.368111 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.368056    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078"} err="failed to get container status \"fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078\": rpc error: code = NotFound desc = could not find container \"fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078\": container with ID starting with fc5e289d66c1f56573cf857e1590ba0aa6a089622faf78ff46543b66b5bc8078 not found: ID does not exist"
Oct 31 01:08:08.368111 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.368072    4935 scope.go:122] "RemoveContainer" containerID="25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066"
Oct 31 01:08:08.368301 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:08:08.368284    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066\": container with ID starting with 25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066 not found: ID does not exist" containerID="25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066"
Oct 31 01:08:08.368340 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.368305    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066"} err="failed to get container status \"25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066\": rpc error: code = NotFound desc = could not find container \"25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066\": container with ID starting with 25a9f46a35ab8ff77717778958c18d26a924a94828890b05094e401fb0b75066 not found: ID does not exist"
Oct 31 01:08:08.513553 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.513527    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"]
Oct 31 01:08:08.560226 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:08.560203    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7f94cf4675-fnhrd"]
Oct 31 01:08:09.943641 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:09.943611    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" path="/var/lib/kubelet/pods/8245e69a-c401-4364-b7d1-b236cba101d0/volumes"
Oct 31 01:08:13.283670 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:13.283636    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"]
Oct 31 01:08:13.284229 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:13.283828    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerName="istio-proxy" containerID="cri-o://2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5" gracePeriod=30
Oct 31 01:08:13.330620 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:13.330590    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"]
Oct 31 01:08:13.330805 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:13.330785    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerName="istio-proxy" containerID="cri-o://6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5" gracePeriod=30
Oct 31 01:08:13.332059 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:13.332025    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"]
Oct 31 01:08:13.332234 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:13.332215    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm" podUID="f12df961-8d85-4e2f-8f62-bc585de0b55d" containerName="istio-proxy" containerID="cri-o://580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a" gracePeriod=30
Oct 31 01:08:14.151699 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:08:14.151663    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:08:18.183506 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.183474    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-s4q4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:08:18.183902 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.183525    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:08:18.469067 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.468534    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-6g8bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:08:18.469067 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.468604    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 01:08:18.600329 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.600303    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:08:18.613942 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.613916    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:08:18.623601 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.623579    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:08:18.678098 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678066    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-envoy\") pod \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") "
Oct 31 01:08:18.678217 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678114    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12df961-8d85-4e2f-8f62-bc585de0b55d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12df961-8d85-4e2f-8f62-bc585de0b55d-istiod-ca-cert\") pod \"f12df961-8d85-4e2f-8f62-bc585de0b55d\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") "
Oct 31 01:08:18.678217 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678137    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istiod-ca-cert\") pod \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") "
Oct 31 01:08:18.678217 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678164    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-podinfo\") pod \"f12df961-8d85-4e2f-8f62-bc585de0b55d\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") "
Oct 31 01:08:18.678217 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678183    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-certs\") pod \"f12df961-8d85-4e2f-8f62-bc585de0b55d\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") "
Oct 31 01:08:18.678473 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678218    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-kube-api-access-vjxvb\" (UniqueName: \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-kube-api-access-vjxvb\") pod \"f12df961-8d85-4e2f-8f62-bc585de0b55d\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") "
Oct 31 01:08:18.678473 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678243    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-podinfo\") pod \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") "
Oct 31 01:08:18.678473 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678283    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-credential-socket\") pod \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") "
Oct 31 01:08:18.678473 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678316    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-token\" (UniqueName: \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-token\") pod \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") "
Oct 31 01:08:18.678473 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678376    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-token\" (UniqueName: \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-token\") pod \"f12df961-8d85-4e2f-8f62-bc585de0b55d\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") "
Oct 31 01:08:18.678473 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678404    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-certs\") pod \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") "
Oct 31 01:08:18.678473 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678447    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-data\") pod \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") "
Oct 31 01:08:18.678816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678478    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-socket\") pod \"f12df961-8d85-4e2f-8f62-bc585de0b55d\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") "
Oct 31 01:08:18.678816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678515    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-credential-socket\") pod \"f12df961-8d85-4e2f-8f62-bc585de0b55d\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") "
Oct 31 01:08:18.678816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678512    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istiod-ca-cert" pod "9ccee86e-ea66-44ba-8fc7-a81cbe82d094" (UID: "9ccee86e-ea66-44ba-8fc7-a81cbe82d094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:18.678816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678548    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-envoy\") pod \"f12df961-8d85-4e2f-8f62-bc585de0b55d\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") "
Oct 31 01:08:18.678816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678591    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-kube-api-access-qh5j8\" (UniqueName: \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-kube-api-access-qh5j8\") pod \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") "
Oct 31 01:08:18.678816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678605    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-certs" pod "f12df961-8d85-4e2f-8f62-bc585de0b55d" (UID: "f12df961-8d85-4e2f-8f62-bc585de0b55d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.678816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678626    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-socket\") pod \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\" (UID: \"9ccee86e-ea66-44ba-8fc7-a81cbe82d094\") "
Oct 31 01:08:18.678816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678631    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12df961-8d85-4e2f-8f62-bc585de0b55d-istiod-ca-cert" pod "f12df961-8d85-4e2f-8f62-bc585de0b55d" (UID: "f12df961-8d85-4e2f-8f62-bc585de0b55d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:18.678816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678689    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-data\") pod \"f12df961-8d85-4e2f-8f62-bc585de0b55d\" (UID: \"f12df961-8d85-4e2f-8f62-bc585de0b55d\") "
Oct 31 01:08:18.679305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678818    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-certs" pod "9ccee86e-ea66-44ba-8fc7-a81cbe82d094" (UID: "9ccee86e-ea66-44ba-8fc7-a81cbe82d094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.679305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678823    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-credential-socket" pod "f12df961-8d85-4e2f-8f62-bc585de0b55d" (UID: "f12df961-8d85-4e2f-8f62-bc585de0b55d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.679305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.678976    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-data" pod "9ccee86e-ea66-44ba-8fc7-a81cbe82d094" (UID: "9ccee86e-ea66-44ba-8fc7-a81cbe82d094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.679305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.679025    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f12df961-8d85-4e2f-8f62-bc585de0b55d-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.679305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.679061    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.679305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.679075    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.679305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.679092    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.679305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.679105    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.679305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.679166    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-socket" pod "f12df961-8d85-4e2f-8f62-bc585de0b55d" (UID: "f12df961-8d85-4e2f-8f62-bc585de0b55d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.679305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.679288    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-socket" pod "9ccee86e-ea66-44ba-8fc7-a81cbe82d094" (UID: "9ccee86e-ea66-44ba-8fc7-a81cbe82d094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.679812 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.679475    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-data" pod "f12df961-8d85-4e2f-8f62-bc585de0b55d" (UID: "f12df961-8d85-4e2f-8f62-bc585de0b55d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.679812 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.679554    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-credential-socket" pod "9ccee86e-ea66-44ba-8fc7-a81cbe82d094" (UID: "9ccee86e-ea66-44ba-8fc7-a81cbe82d094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.680958 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.680933    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-podinfo" pod "f12df961-8d85-4e2f-8f62-bc585de0b55d" (UID: "f12df961-8d85-4e2f-8f62-bc585de0b55d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:08:18.681111 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.681007    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-envoy" pod "9ccee86e-ea66-44ba-8fc7-a81cbe82d094" (UID: "9ccee86e-ea66-44ba-8fc7-a81cbe82d094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.681341 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.681323    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-token" pod "f12df961-8d85-4e2f-8f62-bc585de0b55d" (UID: "f12df961-8d85-4e2f-8f62-bc585de0b55d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:18.681341 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.681332    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-token" pod "9ccee86e-ea66-44ba-8fc7-a81cbe82d094" (UID: "9ccee86e-ea66-44ba-8fc7-a81cbe82d094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:18.681777 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.681761    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-envoy" pod "f12df961-8d85-4e2f-8f62-bc585de0b55d" (UID: "f12df961-8d85-4e2f-8f62-bc585de0b55d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.682106 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.682090    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-podinfo" pod "9ccee86e-ea66-44ba-8fc7-a81cbe82d094" (UID: "9ccee86e-ea66-44ba-8fc7-a81cbe82d094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:08:18.682245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.682224    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-kube-api-access-qh5j8" pod "9ccee86e-ea66-44ba-8fc7-a81cbe82d094" (UID: "9ccee86e-ea66-44ba-8fc7-a81cbe82d094"). InnerVolumeSpecName "kube-api-access-qh5j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:18.682360 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.682310    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-kube-api-access-vjxvb" pod "f12df961-8d85-4e2f-8f62-bc585de0b55d" (UID: "f12df961-8d85-4e2f-8f62-bc585de0b55d"). InnerVolumeSpecName "kube-api-access-vjxvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:18.780106 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780084    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-token\" (UniqueName: \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-token\") pod \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") "
Oct 31 01:08:18.780245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780150    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istiod-ca-cert\") pod \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") "
Oct 31 01:08:18.780245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780168    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-envoy\") pod \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") "
Oct 31 01:08:18.780245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780190    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-credential-socket\") pod \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") "
Oct 31 01:08:18.780245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780211    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-data\") pod \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") "
Oct 31 01:08:18.780245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780235    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-kube-api-access-z4g6l\" (UniqueName: \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-kube-api-access-z4g6l\") pod \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") "
Oct 31 01:08:18.780511 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780263    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-certs\") pod \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") "
Oct 31 01:08:18.780511 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780291    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-podinfo\") pod \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") "
Oct 31 01:08:18.780511 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780317    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-socket\") pod \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\" (UID: \"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff\") "
Oct 31 01:08:18.780671 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780547    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-certs" pod "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" (UID: "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.780671 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780557    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istiod-ca-cert" pod "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" (UID: "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:18.780671 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780600    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780671 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780621    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780671 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780636    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjxvb\" (UniqueName: \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-kube-api-access-vjxvb\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780671 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780653    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780671 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780666    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780671    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-data" pod "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" (UID: "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.780994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780679    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780784    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-credential-socket" pod "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" (UID: "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.780994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780800    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-socket" pod "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" (UID: "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.780994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780723    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780837    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780856    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780869    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780882    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh5j8\" (UniqueName: \"kubernetes.io/projected/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-kube-api-access-qh5j8\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780899    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccee86e-ea66-44ba-8fc7-a81cbe82d094-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.780994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.780920    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f12df961-8d85-4e2f-8f62-bc585de0b55d-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.782367 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.782344    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-kube-api-access-z4g6l" pod "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" (UID: "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff"). InnerVolumeSpecName "kube-api-access-z4g6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:18.782741 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.782719    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-token" pod "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" (UID: "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:18.782996 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.782978    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-envoy" pod "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" (UID: "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:18.783070 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.783002    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-podinfo" pod "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" (UID: "d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:08:18.881477 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.881453    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.881477 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.881476    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.881718 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.881486    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.881718 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.881494    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.881718 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.881503    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4g6l\" (UniqueName: \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-kube-api-access-z4g6l\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.881718 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.881511    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.881718 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.881522    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.881718 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.881535    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:18.881718 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:18.881547    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:19.400864 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.400831    4935 generic.go:356] "Generic (PLEG): container finished" podID="f12df961-8d85-4e2f-8f62-bc585de0b55d" containerID="580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a" exitCode=0
Oct 31 01:08:19.401307 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.400876    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"
Oct 31 01:08:19.401307 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.400920    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm" event={"ID":"f12df961-8d85-4e2f-8f62-bc585de0b55d","Type":"ContainerDied","Data":"580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a"}
Oct 31 01:08:19.401307 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.400969    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm" event={"ID":"f12df961-8d85-4e2f-8f62-bc585de0b55d","Type":"ContainerDied","Data":"4dbc9146900f6949bd60bade5e6ba6cd67e6af266f04b602a6c638788f5f34ef"}
Oct 31 01:08:19.401307 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.400997    4935 scope.go:122] "RemoveContainer" containerID="580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a"
Oct 31 01:08:19.402332 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.402310    4935 generic.go:356] "Generic (PLEG): container finished" podID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerID="6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5" exitCode=0
Oct 31 01:08:19.402448 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.402348    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"
Oct 31 01:08:19.402448 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.402397    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" event={"ID":"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff","Type":"ContainerDied","Data":"6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5"}
Oct 31 01:08:19.402713 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.402457    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v" event={"ID":"d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff","Type":"ContainerDied","Data":"46800759352b89861d7a9acb097bb90f85ee6b7da95fed13692139ca5e0795ec"}
Oct 31 01:08:19.403932 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.403911    4935 generic.go:356] "Generic (PLEG): container finished" podID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerID="2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5" exitCode=0
Oct 31 01:08:19.404050 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.403943    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" event={"ID":"9ccee86e-ea66-44ba-8fc7-a81cbe82d094","Type":"ContainerDied","Data":"2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5"}
Oct 31 01:08:19.404050 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.403961    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"
Oct 31 01:08:19.404050 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.403982    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg" event={"ID":"9ccee86e-ea66-44ba-8fc7-a81cbe82d094","Type":"ContainerDied","Data":"769fe50dd83787683f9a7b4e11b9938b485657f53d98c0785728b0cb2c6b2801"}
Oct 31 01:08:19.410138 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.410120    4935 scope.go:122] "RemoveContainer" containerID="580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a"
Oct 31 01:08:19.410445 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:08:19.410416    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a\": container with ID starting with 580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a not found: ID does not exist" containerID="580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a"
Oct 31 01:08:19.410730 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.410456    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a"} err="failed to get container status \"580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a\": rpc error: code = NotFound desc = could not find container \"580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a\": container with ID starting with 580479f953f7227d4cfbffa86273c0c2a83e43e0fcbee8664cd1f2596769be3a not found: ID does not exist"
Oct 31 01:08:19.410730 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.410478    4935 scope.go:122] "RemoveContainer" containerID="6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5"
Oct 31 01:08:19.419956 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.419939    4935 scope.go:122] "RemoveContainer" containerID="6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5"
Oct 31 01:08:19.420216 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:08:19.420197    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5\": container with ID starting with 6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5 not found: ID does not exist" containerID="6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5"
Oct 31 01:08:19.420263 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.420223    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5"} err="failed to get container status \"6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5\": rpc error: code = NotFound desc = could not find container \"6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5\": container with ID starting with 6b23ddf69a6c2fa3e8bb96dd0357dea74df3c1a3d8bee9585bf675c9de819ab5 not found: ID does not exist"
Oct 31 01:08:19.420263 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.420239    4935 scope.go:122] "RemoveContainer" containerID="2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5"
Oct 31 01:08:19.427852 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.427834    4935 scope.go:122] "RemoveContainer" containerID="2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5"
Oct 31 01:08:19.428107 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:08:19.428090    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5\": container with ID starting with 2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5 not found: ID does not exist" containerID="2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5"
Oct 31 01:08:19.428156 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.428115    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5"} err="failed to get container status \"2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5\": rpc error: code = NotFound desc = could not find container \"2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5\": container with ID starting with 2fd4c177542d5457cba2e77ea27da61b2094d6395cc0ff609de8215b0cb7ecb5 not found: ID does not exist"
Oct 31 01:08:19.563786 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.563731    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"]
Oct 31 01:08:19.610724 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.610696    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-6g8bg"]
Oct 31 01:08:19.677945 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.677922    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"]
Oct 31 01:08:19.684532 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.684510    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-s4q4v"]
Oct 31 01:08:19.791699 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.791676    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"]
Oct 31 01:08:19.797849 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.797830    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vdhnm"]
Oct 31 01:08:19.942568 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.942499    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" path="/var/lib/kubelet/pods/9ccee86e-ea66-44ba-8fc7-a81cbe82d094/volumes"
Oct 31 01:08:19.942958 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.942939    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" path="/var/lib/kubelet/pods/d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff/volumes"
Oct 31 01:08:19.943379 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:19.943362    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12df961-8d85-4e2f-8f62-bc585de0b55d" path="/var/lib/kubelet/pods/f12df961-8d85-4e2f-8f62-bc585de0b55d/volumes"
Oct 31 01:08:44.152066 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:08:44.152008    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:08:48.168613 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.168585    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:08:48.176636 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.176607    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"]
Oct 31 01:08:48.177060 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177023    4935 cpu_manager.go:414] "containerMap: removing container" podUID="f12df961-8d85-4e2f-8f62-bc585de0b55d" containerName="istio-proxy"
Oct 31 01:08:48.177060 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177058    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12df961-8d85-4e2f-8f62-bc585de0b55d" containerName="istio-proxy"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177072    4935 cpu_manager.go:414] "containerMap: removing container" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="storage-initializer"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177078    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="storage-initializer"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177091    4935 cpu_manager.go:414] "containerMap: removing container" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerName="istio-proxy"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177096    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerName="istio-proxy"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177105    4935 cpu_manager.go:414] "containerMap: removing container" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="main"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177109    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="main"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177132    4935 cpu_manager.go:414] "containerMap: removing container" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerName="istio-proxy"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177137    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerName="istio-proxy"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177199    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91cc4b2-3d1b-4ac8-945c-34029d5bf5ff" containerName="istio-proxy"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177213    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12df961-8d85-4e2f-8f62-bc585de0b55d" containerName="istio-proxy"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177224    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccee86e-ea66-44ba-8fc7-a81cbe82d094" containerName="istio-proxy"
Oct 31 01:08:48.177251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.177232    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8245e69a-c401-4364-b7d1-b236cba101d0" containerName="main"
Oct 31 01:08:48.180145 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.180125    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.183774 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.183752    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:08:48.218024 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.218001    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.218157 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.218051    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fb69cdf-13d5-415d-a51a-e0687ba42233-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.218157 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.218128    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.218246 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.218195    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.218246 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.218222    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb69cdf-13d5-415d-a51a-e0687ba42233-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.218246 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.218241    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.218348 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.218291    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.218348 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.218338    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrkxh\" (UniqueName: \"kubernetes.io/projected/7fb69cdf-13d5-415d-a51a-e0687ba42233-kube-api-access-lrkxh\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.277578 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.277540    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Oct 31 01:08:48.277578 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.277559    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Oct 31 01:08:48.277578 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.277570    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:08:48.277829 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.277582    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Oct 31 01:08:48.277829 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.277700    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Oct 31 01:08:48.277829 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.277719    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-n8fmm\""
Oct 31 01:08:48.279003 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.278978    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"]
Oct 31 01:08:48.318840 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.318810    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319029 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.318854    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319029 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.318891    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrkxh\" (UniqueName: \"kubernetes.io/projected/7fb69cdf-13d5-415d-a51a-e0687ba42233-kube-api-access-lrkxh\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319029 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.318946    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319029 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.318980    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fb69cdf-13d5-415d-a51a-e0687ba42233-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319029 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.319004    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319329 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.319083    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319329 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.319122    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb69cdf-13d5-415d-a51a-e0687ba42233-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319329 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.319266    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319329 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.319315    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319542 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.319411    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319597 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.319583    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.319926 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.319873    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fb69cdf-13d5-415d-a51a-e0687ba42233-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.321646 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.321627    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.322024 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.322004    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb69cdf-13d5-415d-a51a-e0687ba42233-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.379280 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.379249    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrkxh\" (UniqueName: \"kubernetes.io/projected/7fb69cdf-13d5-415d-a51a-e0687ba42233-kube-api-access-lrkxh\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.497247 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.497214    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:48.930547 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:48.930471    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"]
Oct 31 01:08:48.936734 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:08:48.936707    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb69cdf_13d5_415d_a51a_e0687ba42233.slice/crio-70b166cdf694dbcb7c83924ccea33471c88075c77f99653e9ea389b0da25d4c2 WatchSource:0}: Error finding container 70b166cdf694dbcb7c83924ccea33471c88075c77f99653e9ea389b0da25d4c2: Status 404 returned error can't find the container with id 70b166cdf694dbcb7c83924ccea33471c88075c77f99653e9ea389b0da25d4c2
Oct 31 01:08:49.540095 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:49.540058    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" event={"ID":"7fb69cdf-13d5-415d-a51a-e0687ba42233","Type":"ContainerStarted","Data":"54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24"}
Oct 31 01:08:49.540481 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:49.540118    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" event={"ID":"7fb69cdf-13d5-415d-a51a-e0687ba42233","Type":"ContainerStarted","Data":"70b166cdf694dbcb7c83924ccea33471c88075c77f99653e9ea389b0da25d4c2"}
Oct 31 01:08:52.554940 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:52.554909    4935 generic.go:356] "Generic (PLEG): container finished" podID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerID="54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24" exitCode=0
Oct 31 01:08:52.555383 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:52.554988    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" event={"ID":"7fb69cdf-13d5-415d-a51a-e0687ba42233","Type":"ContainerDied","Data":"54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24"}
Oct 31 01:08:53.561520 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:53.561489    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" event={"ID":"7fb69cdf-13d5-415d-a51a-e0687ba42233","Type":"ContainerStarted","Data":"f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f"}
Oct 31 01:08:53.681547 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:53.681494    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" podStartSLOduration=2.538027075 podStartE2EDuration="5.681480395s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.14345332s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:53.679654569 +0000 UTC m=+2405.862183863" watchObservedRunningTime="2026-10-31 01:08:53.681480395 +0000 UTC m=+2405.864009707"
Oct 31 01:08:54.498103 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:54.498065    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:58.497483 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:58.497440    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:08:58.499263 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:08:58.499230    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 01:09:08.498010 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:08.497968    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 01:09:14.152481 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:09:14.152441    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:09:18.498571 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:18.498531    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 01:09:28.497921 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:28.497875    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 01:09:28.950590 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:28.950506    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"]
Oct 31 01:09:28.954286 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:28.954270    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:28.997342 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:28.997318    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"]
Oct 31 01:09:29.000593 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.000576    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"]
Oct 31 01:09:29.000699 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.000682    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.003975 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.003954    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.049595 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.049562    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"]
Oct 31 01:09:29.089770 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.089740    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.089919 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.089776    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e466f-5bce-4ba9-9327-681d81a5f0f3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.089919 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.089849    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbmd6\" (UniqueName: \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-kube-api-access-gbmd6\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.089919 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.089881    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.089919 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.089915    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.090097 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.089939    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.090097 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.090055    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.090172 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.090156    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.090204 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.090194    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.096364 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.096339    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"]
Oct 31 01:09:29.099531 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.099507    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"]
Oct 31 01:09:29.190916 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.190885    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.191079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.190932    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d62ef7-c133-4cdd-84de-14d4a0acb945-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.191079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.190970    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.191079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.190997    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.191079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191022    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.191255 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191138    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.191255 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191177    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.191255 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191240    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.191370 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191320    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.191370 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191359    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.191483 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191411    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.191483 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191445    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.191588 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191521    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.191588 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191549    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.191787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191587    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.191787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191736    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9v6g\" (UniqueName: \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-kube-api-access-s9v6g\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.191787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191769    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.191787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191789    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.191787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191796    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.191787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191814    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.191787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191841    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e466f-5bce-4ba9-9327-681d81a5f0f3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.192251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191892    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnsxg\" (UniqueName: \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-kube-api-access-pnsxg\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.192251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191923    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbmd6\" (UniqueName: \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-kube-api-access-gbmd6\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.192251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191959    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.192251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.191984    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.192251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.192012    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.192251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.192019    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.192251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.192093    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.192251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.192128    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.192251 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.192152    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.192696 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.192296    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.192696 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.192660    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e466f-5bce-4ba9-9327-681d81a5f0f3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.194207 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.194178    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.195373 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.195346    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.251423 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.251400    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.251699 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.251676    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbmd6\" (UniqueName: \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-kube-api-access-gbmd6\") pod \"openshift-ai-inference-openshift-default-568c495856-nfb67\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.267261 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.267241    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:29.293070 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293023    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.293202 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293084    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.293202 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293104    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.293202 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293121    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.293202 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293136    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.293202 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293155    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.293202 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293186    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.293512 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293221    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.293512 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293248    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.293512 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293270    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.293512 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293285    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.293512 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293343    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9v6g\" (UniqueName: \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-kube-api-access-s9v6g\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.293512 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293371    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.293512 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293412    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnsxg\" (UniqueName: \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-kube-api-access-pnsxg\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.293512 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293446    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.293512 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293454    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.293512 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293495    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.294014 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293520    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.294014 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293543    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.294014 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293608    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d62ef7-c133-4cdd-84de-14d4a0acb945-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.294014 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.293988    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.294294 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.294272    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.294365 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.294286    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.294365 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.294321    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d62ef7-c133-4cdd-84de-14d4a0acb945-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.294948 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.294561    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.294948 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.294782    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.294948 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.294912    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.295263 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.295243    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.295858 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.295834    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.297190 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.297159    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.297404 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.297379    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.298912 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.298892    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.355335 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.354826    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.355498 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.355424    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.355498 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.355466    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnsxg\" (UniqueName: \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-kube-api-access-pnsxg\") pod \"openshift-ai-inference-openshift-default-568c495856-b9b65\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.355879 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.355817    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9v6g\" (UniqueName: \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-kube-api-access-s9v6g\") pod \"openshift-ai-inference-openshift-default-568c495856-kzxcm\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.615008 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.614933    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:29.621393 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.621367    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:29.646152 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.646121    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"]
Oct 31 01:09:29.651808 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:09:29.651777    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608e466f_5bce_4ba9_9327_681d81a5f0f3.slice/crio-467eb655c16687d3bc66a7561adb24f2d6029112a4d9f02ab5eacad98b193cfb WatchSource:0}: Error finding container 467eb655c16687d3bc66a7561adb24f2d6029112a4d9f02ab5eacad98b193cfb: Status 404 returned error can't find the container with id 467eb655c16687d3bc66a7561adb24f2d6029112a4d9f02ab5eacad98b193cfb
Oct 31 01:09:29.654519 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.654475    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:29.654605 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.654571    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:29.654646 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.654618    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:29.744897 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:29.744864    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" event={"ID":"608e466f-5bce-4ba9-9327-681d81a5f0f3","Type":"ContainerStarted","Data":"467eb655c16687d3bc66a7561adb24f2d6029112a4d9f02ab5eacad98b193cfb"}
Oct 31 01:09:30.000943 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.000903    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"]
Oct 31 01:09:30.007536 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:09:30.007503    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d08047_d9b7_4d05_a5da_e6b97b03b36e.slice/crio-fd1620eff57a8ec2efb379380bac29f16bcb63f0ab8fb7938b480e31701a7bc1 WatchSource:0}: Error finding container fd1620eff57a8ec2efb379380bac29f16bcb63f0ab8fb7938b480e31701a7bc1: Status 404 returned error can't find the container with id fd1620eff57a8ec2efb379380bac29f16bcb63f0ab8fb7938b480e31701a7bc1
Oct 31 01:09:30.009654 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.009622    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:30.009735 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.009695    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:30.009735 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.009723    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:30.052290 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.052260    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"]
Oct 31 01:09:30.059015 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:09:30.058973    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d62ef7_c133_4cdd_84de_14d4a0acb945.slice/crio-11b80ad6f99182026310d9c83f3cab53641c0dbdb3dc7817b9c4be6771a9d02d WatchSource:0}: Error finding container 11b80ad6f99182026310d9c83f3cab53641c0dbdb3dc7817b9c4be6771a9d02d: Status 404 returned error can't find the container with id 11b80ad6f99182026310d9c83f3cab53641c0dbdb3dc7817b9c4be6771a9d02d
Oct 31 01:09:30.061419 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.061384    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:30.061505 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.061460    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:30.061505 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.061491    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:09:30.770061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.759768    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" event={"ID":"f4d08047-d9b7-4d05-a5da-e6b97b03b36e","Type":"ContainerStarted","Data":"071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb"}
Oct 31 01:09:30.770061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.759833    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" event={"ID":"f4d08047-d9b7-4d05-a5da-e6b97b03b36e","Type":"ContainerStarted","Data":"fd1620eff57a8ec2efb379380bac29f16bcb63f0ab8fb7938b480e31701a7bc1"}
Oct 31 01:09:30.770061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.763437    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" event={"ID":"608e466f-5bce-4ba9-9327-681d81a5f0f3","Type":"ContainerStarted","Data":"b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab"}
Oct 31 01:09:30.784839 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.784077    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" event={"ID":"06d62ef7-c133-4cdd-84de-14d4a0acb945","Type":"ContainerStarted","Data":"377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125"}
Oct 31 01:09:30.784839 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:30.784150    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" event={"ID":"06d62ef7-c133-4cdd-84de-14d4a0acb945","Type":"ContainerStarted","Data":"11b80ad6f99182026310d9c83f3cab53641c0dbdb3dc7817b9c4be6771a9d02d"}
Oct 31 01:09:31.008759 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.008684    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" podStartSLOduration=3.008662773 podStartE2EDuration="3.008662773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:30.887462233 +0000 UTC m=+2443.069991527" watchObservedRunningTime="2026-10-31 01:09:31.008662773 +0000 UTC m=+2443.191192067"
Oct 31 01:09:31.137596 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.137480    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" podStartSLOduration=3.137462008 podStartE2EDuration="3.137462008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:31.135353292 +0000 UTC m=+2443.317882587" watchObservedRunningTime="2026-10-31 01:09:31.137462008 +0000 UTC m=+2443.319991303"
Oct 31 01:09:31.138633 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.138588    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" podStartSLOduration=3.138576105 podStartE2EDuration="3.138576105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:31.009926166 +0000 UTC m=+2443.192455461" watchObservedRunningTime="2026-10-31 01:09:31.138576105 +0000 UTC m=+2443.321105400"
Oct 31 01:09:31.271621 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.271587    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:31.271948 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.271928    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-nfb67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:09:31.272118 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.272093    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" podUID="608e466f-5bce-4ba9-9327-681d81a5f0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:09:31.622063 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.618014    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:31.622447 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.622407    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kzxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:09:31.622625 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.622600    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" podUID="06d62ef7-c133-4cdd-84de-14d4a0acb945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 31 01:09:31.624474 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.624396    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:31.624715 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.624699    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-b9b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:09:31.624864 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:31.624836    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" podUID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 01:09:32.268312 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:32.268277    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-nfb67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:09:32.268904 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:32.268871    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" podUID="608e466f-5bce-4ba9-9327-681d81a5f0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:09:32.616122 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:32.615475    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kzxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:09:32.616122 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:32.615540    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" podUID="06d62ef7-c133-4cdd-84de-14d4a0acb945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 31 01:09:32.624419 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:32.624384    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-b9b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:09:32.624694 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:32.624664    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" podUID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 01:09:33.271542 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:33.271504    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-nfb67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:09:33.272536 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:33.272117    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" podUID="608e466f-5bce-4ba9-9327-681d81a5f0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:09:33.615927 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:33.615419    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kzxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:09:33.615927 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:33.615483    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" podUID="06d62ef7-c133-4cdd-84de-14d4a0acb945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 31 01:09:33.622471 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:33.622356    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-b9b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:09:33.622471 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:33.622415    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" podUID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 01:09:34.271520 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:34.271481    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-nfb67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:09:34.271934 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:34.271903    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" podUID="608e466f-5bce-4ba9-9327-681d81a5f0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:09:34.618626 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:34.618526    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kzxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:09:34.618907 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:34.618872    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" podUID="06d62ef7-c133-4cdd-84de-14d4a0acb945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 31 01:09:34.624435 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:34.624403    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-b9b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:09:34.624686 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:34.624659    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" podUID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 01:09:35.273502 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:35.273461    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-nfb67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:09:35.274155 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:35.274121    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" podUID="608e466f-5bce-4ba9-9327-681d81a5f0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:09:35.615834 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:35.615722    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kzxcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:09:35.615834 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:35.615803    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" podUID="06d62ef7-c133-4cdd-84de-14d4a0acb945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 31 01:09:35.622171 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:35.622142    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-b9b65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:09:35.622389 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:35.622362    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" podUID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 31 01:09:36.271399 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:36.271366    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:36.271722 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:36.271698    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:36.272330 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:36.272313    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:09:36.620914 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:36.620865    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:36.621388 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:36.621370    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:36.622135 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:36.622110    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:09:36.626254 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:36.626231    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:36.626597 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:36.626581    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:36.627450 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:36.627428    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:09:38.498089 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:38.498031    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 01:09:44.153233 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:09:44.153193    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:09:48.498058 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:48.497999    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 01:09:58.498066 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:09:58.497953    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 01:10:08.507517 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:08.507482    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:10:08.514794 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:08.514770    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:10:14.152108 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:10:14.152075    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:10:16.481935 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:16.481901    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"]
Oct 31 01:10:31.510945 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:31.510895    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="main" containerID="cri-o://f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f" gracePeriod=45
Oct 31 01:10:32.061461 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.061435    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:10:32.108938 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.108908    4935 generic.go:356] "Generic (PLEG): container finished" podID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerID="f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f" exitCode=0
Oct 31 01:10:32.109091 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.108959    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"
Oct 31 01:10:32.109091 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.108989    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" event={"ID":"7fb69cdf-13d5-415d-a51a-e0687ba42233","Type":"ContainerDied","Data":"f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f"}
Oct 31 01:10:32.109091 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.109061    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b" event={"ID":"7fb69cdf-13d5-415d-a51a-e0687ba42233","Type":"ContainerDied","Data":"70b166cdf694dbcb7c83924ccea33471c88075c77f99653e9ea389b0da25d4c2"}
Oct 31 01:10:32.109205 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.109091    4935 scope.go:122] "RemoveContainer" containerID="f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f"
Oct 31 01:10:32.117137 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.117110    4935 scope.go:122] "RemoveContainer" containerID="54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24"
Oct 31 01:10:32.125099 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.125084    4935 scope.go:122] "RemoveContainer" containerID="f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f"
Oct 31 01:10:32.125378 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:10:32.125329    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f\": container with ID starting with f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f not found: ID does not exist" containerID="f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f"
Oct 31 01:10:32.125378 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.125358    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f"} err="failed to get container status \"f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f\": rpc error: code = NotFound desc = could not find container \"f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f\": container with ID starting with f4af1deee6c68fb3c96ebeca449f85aad7430569f726d5742e46b89368f3926f not found: ID does not exist"
Oct 31 01:10:32.125378 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.125375    4935 scope.go:122] "RemoveContainer" containerID="54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24"
Oct 31 01:10:32.125602 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:10:32.125585    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24\": container with ID starting with 54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24 not found: ID does not exist" containerID="54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24"
Oct 31 01:10:32.125642 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.125608    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24"} err="failed to get container status \"54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24\": rpc error: code = NotFound desc = could not find container \"54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24\": container with ID starting with 54563cfa1e14df78e0a5a9a6eb82d75ffae68e1e33f5fd6d2a96899336ae4c24 not found: ID does not exist"
Oct 31 01:10:32.187862 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.187837    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-model-cache\") pod \"7fb69cdf-13d5-415d-a51a-e0687ba42233\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") "
Oct 31 01:10:32.187957 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.187873    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-dshm\") pod \"7fb69cdf-13d5-415d-a51a-e0687ba42233\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") "
Oct 31 01:10:32.187957 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.187899    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb69cdf-13d5-415d-a51a-e0687ba42233-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fb69cdf-13d5-415d-a51a-e0687ba42233-cabundle-cert\") pod \"7fb69cdf-13d5-415d-a51a-e0687ba42233\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") "
Oct 31 01:10:32.187957 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.187920    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-kserve-provision-location\") pod \"7fb69cdf-13d5-415d-a51a-e0687ba42233\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") "
Oct 31 01:10:32.188114 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.187991    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-tmp-dir\") pod \"7fb69cdf-13d5-415d-a51a-e0687ba42233\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") "
Oct 31 01:10:32.188114 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.188053    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb69cdf-13d5-415d-a51a-e0687ba42233-tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb69cdf-13d5-415d-a51a-e0687ba42233-tls-certs\") pod \"7fb69cdf-13d5-415d-a51a-e0687ba42233\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") "
Oct 31 01:10:32.188114 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.188092    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-model-cache" pod "7fb69cdf-13d5-415d-a51a-e0687ba42233" (UID: "7fb69cdf-13d5-415d-a51a-e0687ba42233"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:32.188268 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.188106    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb69cdf-13d5-415d-a51a-e0687ba42233-kube-api-access-lrkxh\" (UniqueName: \"kubernetes.io/projected/7fb69cdf-13d5-415d-a51a-e0687ba42233-kube-api-access-lrkxh\") pod \"7fb69cdf-13d5-415d-a51a-e0687ba42233\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") "
Oct 31 01:10:32.188268 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.188178    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-home\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-home\") pod \"7fb69cdf-13d5-415d-a51a-e0687ba42233\" (UID: \"7fb69cdf-13d5-415d-a51a-e0687ba42233\") "
Oct 31 01:10:32.188374 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.188269    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb69cdf-13d5-415d-a51a-e0687ba42233-cabundle-cert" pod "7fb69cdf-13d5-415d-a51a-e0687ba42233" (UID: "7fb69cdf-13d5-415d-a51a-e0687ba42233"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:32.188453 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.188417    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-tmp-dir" pod "7fb69cdf-13d5-415d-a51a-e0687ba42233" (UID: "7fb69cdf-13d5-415d-a51a-e0687ba42233"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:32.188624 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.188559    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fb69cdf-13d5-415d-a51a-e0687ba42233-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:32.188624 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.188585    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:32.188624 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.188599    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:32.188923 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.188902    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-home" pod "7fb69cdf-13d5-415d-a51a-e0687ba42233" (UID: "7fb69cdf-13d5-415d-a51a-e0687ba42233"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:32.190402 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.190375    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb69cdf-13d5-415d-a51a-e0687ba42233-kube-api-access-lrkxh" pod "7fb69cdf-13d5-415d-a51a-e0687ba42233" (UID: "7fb69cdf-13d5-415d-a51a-e0687ba42233"). InnerVolumeSpecName "kube-api-access-lrkxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:32.190502 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.190403    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb69cdf-13d5-415d-a51a-e0687ba42233-tls-certs" pod "7fb69cdf-13d5-415d-a51a-e0687ba42233" (UID: "7fb69cdf-13d5-415d-a51a-e0687ba42233"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:32.190502 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.190425    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-dshm" pod "7fb69cdf-13d5-415d-a51a-e0687ba42233" (UID: "7fb69cdf-13d5-415d-a51a-e0687ba42233"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:32.210331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.210305    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-kserve-provision-location" pod "7fb69cdf-13d5-415d-a51a-e0687ba42233" (UID: "7fb69cdf-13d5-415d-a51a-e0687ba42233"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:32.289414 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.289390    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb69cdf-13d5-415d-a51a-e0687ba42233-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:32.289414 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.289416    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrkxh\" (UniqueName: \"kubernetes.io/projected/7fb69cdf-13d5-415d-a51a-e0687ba42233-kube-api-access-lrkxh\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:32.289564 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.289426    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:32.289564 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.289433    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:32.289564 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.289442    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fb69cdf-13d5-415d-a51a-e0687ba42233-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:10:32.527655 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.527620    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"]
Oct 31 01:10:32.534187 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:32.534156    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-655dd77cf8-ltq9b"]
Oct 31 01:10:33.943490 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:10:33.943455    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" path="/var/lib/kubelet/pods/7fb69cdf-13d5-415d-a51a-e0687ba42233/volumes"
Oct 31 01:10:44.151841 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:10:44.151802    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:11:14.152784 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:11:14.152746    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:11:41.674249 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.674217    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"]
Oct 31 01:11:41.674646 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.674618    4935 cpu_manager.go:414] "containerMap: removing container" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="main"
Oct 31 01:11:41.674646 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.674631    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="main"
Oct 31 01:11:41.674718 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.674647    4935 cpu_manager.go:414] "containerMap: removing container" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="storage-initializer"
Oct 31 01:11:41.674718 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.674655    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="storage-initializer"
Oct 31 01:11:41.674718 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.674712    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb69cdf-13d5-415d-a51a-e0687ba42233" containerName="main"
Oct 31 01:11:41.677711 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.677694    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.729055 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.729012    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Oct 31 01:11:41.761015 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.760992    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Oct 31 01:11:41.761015 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.761011    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:11:41.774769 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.774750    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-76pk2\""
Oct 31 01:11:41.777839 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.777814    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"]
Oct 31 01:11:41.856331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.856299    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-certs\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.856492 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.856350    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-istio-token\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.856492 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.856369    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6edf0352-997e-4594-bb6a-78932b363339-istio-podinfo\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.856571 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.856493    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.856571 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.856519    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-data\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.856571 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.856540    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqkwx\" (UniqueName: \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-kube-api-access-bqkwx\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.856660 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.856604    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-envoy\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.856660 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.856633    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-credential-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.856722 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.856704    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6edf0352-997e-4594-bb6a-78932b363339-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.957942 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.957868    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-certs\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.957942 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.957922    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-istio-token\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958198 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.957945    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6edf0352-997e-4594-bb6a-78932b363339-istio-podinfo\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958198 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.957976    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958198 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.957997    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-data\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958198 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.958023    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqkwx\" (UniqueName: \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-kube-api-access-bqkwx\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958198 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.958080    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-envoy\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958198 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.958104    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-credential-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958198 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.958133    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6edf0352-997e-4594-bb6a-78932b363339-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958566 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.958330    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-certs\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958566 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.958405    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-data\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958673 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.958592    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-credential-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958673 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.958601    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.958938 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.958918    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6edf0352-997e-4594-bb6a-78932b363339-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.960409 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.960387    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-envoy\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:41.960562 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:41.960544    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6edf0352-997e-4594-bb6a-78932b363339-istio-podinfo\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:42.017502 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:42.017479    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-istio-token\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:42.017696 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:42.017676    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqkwx\" (UniqueName: \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-kube-api-access-bqkwx\") pod \"router-gateway-1-openshift-default-56d96d6b69-hljqc\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:42.290257 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:42.290224    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:42.716108 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:42.716023    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"]
Oct 31 01:11:42.722284 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:11:42.722256    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edf0352_997e_4594_bb6a_78932b363339.slice/crio-b146705c8a4d223ebc8225f4337afc6e63516cb0a5e152f03215217a651e3dae WatchSource:0}: Error finding container b146705c8a4d223ebc8225f4337afc6e63516cb0a5e152f03215217a651e3dae: Status 404 returned error can't find the container with id b146705c8a4d223ebc8225f4337afc6e63516cb0a5e152f03215217a651e3dae
Oct 31 01:11:42.724085 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:42.724051    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:42.724153 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:42.724119    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:42.724153 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:42.724149    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:43.443435 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:43.443396    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" event={"ID":"6edf0352-997e-4594-bb6a-78932b363339","Type":"ContainerStarted","Data":"d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384"}
Oct 31 01:11:43.443606 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:43.443458    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" event={"ID":"6edf0352-997e-4594-bb6a-78932b363339","Type":"ContainerStarted","Data":"b146705c8a4d223ebc8225f4337afc6e63516cb0a5e152f03215217a651e3dae"}
Oct 31 01:11:43.614662 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:43.614608    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" podStartSLOduration=2.614594821 podStartE2EDuration="2.614594821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:43.61271283 +0000 UTC m=+2575.795242124" watchObservedRunningTime="2026-10-31 01:11:43.614594821 +0000 UTC m=+2575.797124109"
Oct 31 01:11:44.151989 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:11:44.151949    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:11:44.290524 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:44.290496    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:44.295438 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:44.295418    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:44.448793 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:44.448742    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:44.449469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:44.449454    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:47.895894 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:47.895865    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"]
Oct 31 01:11:47.899690 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:47.899667    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:47.996872 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:47.996841    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Oct 31 01:11:47.996872 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:47.996859    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Oct 31 01:11:47.996872 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:47.996841    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:11:47.997144 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:47.996914    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-qcrbp\""
Oct 31 01:11:47.998736 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:47.998718    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"]
Oct 31 01:11:48.009559 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.009537    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67tsz\" (UniqueName: \"kubernetes.io/projected/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kube-api-access-67tsz\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.009679 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.009574    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-home\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.009679 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.009596    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kserve-provision-location\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.009759 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.009729    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-model-cache\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.009793 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.009766    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tmp-dir\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.009829 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.009796    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tls-certs\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.009829 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.009825    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-cabundle-cert\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.009931 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.009906    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-dshm\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.111232 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.111190    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tmp-dir\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.111438 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.111245    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tls-certs\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.111438 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.111281    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-cabundle-cert\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.111438 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.111298    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-dshm\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.111438 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.111334    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67tsz\" (UniqueName: \"kubernetes.io/projected/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kube-api-access-67tsz\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.111438 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.111379    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-home\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.111438 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.111411    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kserve-provision-location\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.111753 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.111480    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-model-cache\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.111753 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.111671    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tmp-dir\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.111832 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.111804    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-home\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.111961 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.111940    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kserve-provision-location\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.112097 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.112068    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-model-cache\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.112223 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.112164    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-cabundle-cert\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.113765 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.113748    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-dshm\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.114016 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.113997    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tls-certs\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.183915 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.183839    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67tsz\" (UniqueName: \"kubernetes.io/projected/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kube-api-access-67tsz\") pod \"router-with-refs-test-kserve-6c5b97b99b-j9cjq\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.215749 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.215716    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:48.723462 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:48.723430    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"]
Oct 31 01:11:48.728740 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:11:48.728711    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6011a2_7b15_456e_b0f0_43f8d7cb0d7a.slice/crio-2f53011aaefee2ef26c87d2a55c80a10789b10cb44a7e5e1abc9b57fee74303a WatchSource:0}: Error finding container 2f53011aaefee2ef26c87d2a55c80a10789b10cb44a7e5e1abc9b57fee74303a: Status 404 returned error can't find the container with id 2f53011aaefee2ef26c87d2a55c80a10789b10cb44a7e5e1abc9b57fee74303a
Oct 31 01:11:49.492904 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:49.492853    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" event={"ID":"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a","Type":"ContainerStarted","Data":"79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69"}
Oct 31 01:11:49.492904 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:49.492912    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" event={"ID":"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a","Type":"ContainerStarted","Data":"2f53011aaefee2ef26c87d2a55c80a10789b10cb44a7e5e1abc9b57fee74303a"}
Oct 31 01:11:51.505391 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:51.505360    4935 generic.go:356] "Generic (PLEG): container finished" podID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerID="79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69" exitCode=0
Oct 31 01:11:51.505772 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:51.505431    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" event={"ID":"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a","Type":"ContainerDied","Data":"79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69"}
Oct 31 01:11:52.512109 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:52.512066    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" event={"ID":"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a","Type":"ContainerStarted","Data":"4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e"}
Oct 31 01:11:52.724920 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:52.724852    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podStartSLOduration=3.254138922 podStartE2EDuration="5.724836607s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.470697685s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:52.676650369 +0000 UTC m=+2584.859179665" watchObservedRunningTime="2026-10-31 01:11:52.724836607 +0000 UTC m=+2584.907365900"
Oct 31 01:11:53.216402 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:53.216355    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:55.528179 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:55.528143    4935 generic.go:356] "Generic (PLEG): container finished" podID="6edf0352-997e-4594-bb6a-78932b363339" containerID="d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384" exitCode=0
Oct 31 01:11:55.528758 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:55.528227    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" event={"ID":"6edf0352-997e-4594-bb6a-78932b363339","Type":"ContainerDied","Data":"d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384"}
Oct 31 01:11:55.528758 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:55.528607    4935 scope.go:122] "RemoveContainer" containerID="d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384"
Oct 31 01:11:55.529702 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:55.529671    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:55.529791 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:55.529774    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:55.529852 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:55.529815    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:11:56.535303 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:56.535267    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" event={"ID":"6edf0352-997e-4594-bb6a-78932b363339","Type":"ContainerStarted","Data":"45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2"}
Oct 31 01:11:57.290388 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:57.290353    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:11:57.291615 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:57.291586    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:11:58.216373 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:58.216334    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:11:58.218091 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:58.218062    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 31 01:11:58.290756 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:58.290712    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:11:59.291602 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:59.291551    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:11:59.448796 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:11:59.448756    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:12:00.291561 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:12:00.291511    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:12:01.291304 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:12:01.291264    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:12:02.295626 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:12:02.295590    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:12:02.296690 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:12:02.296666    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:12:08.216389 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:12:08.216339    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 31 01:12:14.152682 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:12:14.152643    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:12:18.216441 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:12:18.216401    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 31 01:12:28.216448 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:12:28.216407    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 31 01:12:38.216305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:12:38.216254    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 31 01:12:44.152965 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:12:44.152924    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:12:48.217087 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:12:48.217016    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 31 01:12:58.216469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:12:58.216369    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 31 01:13:08.217095 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:08.217027    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 31 01:13:14.153013 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:13:14.152969    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:13:18.217008 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:18.216956    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 31 01:13:24.416731 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.416693    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"]
Oct 31 01:13:24.421398 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.421382    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.424405 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.424385    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-home\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.424520 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.424411    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dcd9130-8332-4efd-b5a0-2212f59ec374-cabundle-cert\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.424520 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.424436    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98kvj\" (UniqueName: \"kubernetes.io/projected/0dcd9130-8332-4efd-b5a0-2212f59ec374-kube-api-access-98kvj\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.424636 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.424599    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-dshm\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.424693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.424633    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-tmp-dir\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.424754 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.424720    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcd9130-8332-4efd-b5a0-2212f59ec374-tls-certs\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.424812 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.424787    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-model-cache\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.424865 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.424815    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-kserve-provision-location\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.473197 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.473162    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"]
Oct 31 01:13:24.516972 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.516936    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-jf94k\""
Oct 31 01:13:24.517181 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.516976    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Oct 31 01:13:24.517181 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.517019    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Oct 31 01:13:24.517181 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.517019    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:13:24.517181 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.517084    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Oct 31 01:13:24.517181 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.517112    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Oct 31 01:13:24.525824 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.525798    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-dshm\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.525890 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.525843    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-tmp-dir\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.525945 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.525902    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcd9130-8332-4efd-b5a0-2212f59ec374-tls-certs\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.525998 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.525951    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-model-cache\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.525998 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.525980    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-kserve-provision-location\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.526142 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.526056    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-home\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.526142 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.526088    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dcd9130-8332-4efd-b5a0-2212f59ec374-cabundle-cert\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.526142 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.526136    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98kvj\" (UniqueName: \"kubernetes.io/projected/0dcd9130-8332-4efd-b5a0-2212f59ec374-kube-api-access-98kvj\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.526403 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.526356    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-tmp-dir\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.526626 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.526599    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-kserve-provision-location\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.526719 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.526604    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-model-cache\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.526944 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.526727    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-home\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.527000 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.526964    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dcd9130-8332-4efd-b5a0-2212f59ec374-cabundle-cert\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.528531 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.528508    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-dshm\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.528798 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.528779    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcd9130-8332-4efd-b5a0-2212f59ec374-tls-certs\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.589079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.589031    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98kvj\" (UniqueName: \"kubernetes.io/projected/0dcd9130-8332-4efd-b5a0-2212f59ec374-kube-api-access-98kvj\") pod \"stop-feature-test-kserve-5678d64c4-ssp6q\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:24.739067 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:24.739010    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:25.174725 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:25.174633    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"]
Oct 31 01:13:25.181239 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:13:25.181207    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcd9130_8332_4efd_b5a0_2212f59ec374.slice/crio-b2273cae772aa56a97ce6ae569e5f237df0a5d72d5c9871eb94b5aaaa83df11d WatchSource:0}: Error finding container b2273cae772aa56a97ce6ae569e5f237df0a5d72d5c9871eb94b5aaaa83df11d: Status 404 returned error can't find the container with id b2273cae772aa56a97ce6ae569e5f237df0a5d72d5c9871eb94b5aaaa83df11d
Oct 31 01:13:25.990714 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:25.990679    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" event={"ID":"0dcd9130-8332-4efd-b5a0-2212f59ec374","Type":"ContainerStarted","Data":"7e924e5bed4d6402ce65056ab612dc74dbbef7b428d293f83a9f7c8d21d2c35c"}
Oct 31 01:13:25.991239 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:25.990731    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" event={"ID":"0dcd9130-8332-4efd-b5a0-2212f59ec374","Type":"ContainerStarted","Data":"b2273cae772aa56a97ce6ae569e5f237df0a5d72d5c9871eb94b5aaaa83df11d"}
Oct 31 01:13:28.004181 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:28.004149    4935 generic.go:356] "Generic (PLEG): container finished" podID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerID="7e924e5bed4d6402ce65056ab612dc74dbbef7b428d293f83a9f7c8d21d2c35c" exitCode=0
Oct 31 01:13:28.004581 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:28.004223    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" event={"ID":"0dcd9130-8332-4efd-b5a0-2212f59ec374","Type":"ContainerDied","Data":"7e924e5bed4d6402ce65056ab612dc74dbbef7b428d293f83a9f7c8d21d2c35c"}
Oct 31 01:13:28.216660 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:28.216618    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 31 01:13:29.011225 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:29.011185    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" event={"ID":"0dcd9130-8332-4efd-b5a0-2212f59ec374","Type":"ContainerStarted","Data":"0a8a4a9aa2d041beb90fac24917a0fe5e25f6114049a33addd3280b0eec2e14b"}
Oct 31 01:13:29.224610 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:29.224542    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podStartSLOduration=3.026525944 podStartE2EDuration="5.2245267s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.198000756s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:29.222976003 +0000 UTC m=+2681.405505298" watchObservedRunningTime="2026-10-31 01:13:29.2245267 +0000 UTC m=+2681.407055993"
Oct 31 01:13:29.740161 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:29.740125    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:34.739390 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:34.739356    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:13:34.740794 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:34.740769    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 31 01:13:38.226302 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:38.226264    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:13:38.233687 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:38.233661    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:13:44.155861 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:13:44.155817    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:13:44.739742 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:44.739701    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 31 01:13:46.294749 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:46.294718    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"]
Oct 31 01:13:48.200415 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:48.200384    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:13:48.222255 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:48.222230    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:13:54.739683 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:13:54.739639    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 31 01:14:01.319831 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:01.319757    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main" containerID="cri-o://4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e" gracePeriod=45
Oct 31 01:14:02.069177 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.069150    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:14:02.116068 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.115972    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kserve-provision-location\") pod \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") "
Oct 31 01:14:02.116223 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.116074    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kube-api-access-67tsz\" (UniqueName: \"kubernetes.io/projected/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kube-api-access-67tsz\") pod \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") "
Oct 31 01:14:02.116223 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.116112    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tmp-dir\") pod \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") "
Oct 31 01:14:02.116223 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.116151    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-cabundle-cert\") pod \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") "
Oct 31 01:14:02.116223 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.116216    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tls-certs\") pod \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") "
Oct 31 01:14:02.116434 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.116243    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-dshm\") pod \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") "
Oct 31 01:14:02.116434 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.116279    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-model-cache\") pod \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") "
Oct 31 01:14:02.116434 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.116320    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-home\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-home\") pod \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\" (UID: \"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a\") "
Oct 31 01:14:02.117065 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.116635    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-cabundle-cert" pod "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" (UID: "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:02.117452 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.117206    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-model-cache" pod "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" (UID: "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:02.117452 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.117439    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-home" pod "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" (UID: "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:02.118738 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.118714    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-dshm" pod "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" (UID: "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:02.118955 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.118934    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kube-api-access-67tsz" pod "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" (UID: "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a"). InnerVolumeSpecName "kube-api-access-67tsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:02.119500 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.119468    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tls-certs" pod "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" (UID: "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:14:02.128069 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.128011    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tmp-dir" pod "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" (UID: "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:02.149254 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.149224    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kserve-provision-location" pod "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" (UID: "0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:02.208412 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.208377    4935 generic.go:356] "Generic (PLEG): container finished" podID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerID="4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e" exitCode=0
Oct 31 01:14:02.208552 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.208437    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"
Oct 31 01:14:02.208552 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.208451    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" event={"ID":"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a","Type":"ContainerDied","Data":"4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e"}
Oct 31 01:14:02.208552 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.208509    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq" event={"ID":"0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a","Type":"ContainerDied","Data":"2f53011aaefee2ef26c87d2a55c80a10789b10cb44a7e5e1abc9b57fee74303a"}
Oct 31 01:14:02.208552 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.208536    4935 scope.go:122] "RemoveContainer" containerID="4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e"
Oct 31 01:14:02.217287 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.217259    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:02.217287 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.217285    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:02.217461 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.217301    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:02.217461 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.217313    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:02.217461 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.217331    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:02.217461 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.217345    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67tsz\" (UniqueName: \"kubernetes.io/projected/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-kube-api-access-67tsz\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:02.217461 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.217357    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:02.217461 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.217370    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:02.218489 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.218474    4935 scope.go:122] "RemoveContainer" containerID="79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69"
Oct 31 01:14:02.228314 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.228294    4935 scope.go:122] "RemoveContainer" containerID="4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e"
Oct 31 01:14:02.228595 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:14:02.228573    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e\": container with ID starting with 4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e not found: ID does not exist" containerID="4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e"
Oct 31 01:14:02.228658 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.228603    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e"} err="failed to get container status \"4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e\": rpc error: code = NotFound desc = could not find container \"4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e\": container with ID starting with 4f18574f85d42ef2f2f9f41d9047ef1c6639f6a4e16be8bf5d204b2c0c891d6e not found: ID does not exist"
Oct 31 01:14:02.228658 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.228620    4935 scope.go:122] "RemoveContainer" containerID="79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69"
Oct 31 01:14:02.228865 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:14:02.228847    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69\": container with ID starting with 79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69 not found: ID does not exist" containerID="79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69"
Oct 31 01:14:02.228915 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.228878    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69"} err="failed to get container status \"79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69\": rpc error: code = NotFound desc = could not find container \"79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69\": container with ID starting with 79440cb7c53b46928859dc1ecbc24a174340023c9ebef6d58afccc36d378ac69 not found: ID does not exist"
Oct 31 01:14:02.344335 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.344302    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"]
Oct 31 01:14:02.352855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:02.352826    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6c5b97b99b-j9cjq"]
Oct 31 01:14:03.943915 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:03.943878    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" path="/var/lib/kubelet/pods/0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a/volumes"
Oct 31 01:14:04.739831 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:04.739787    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 31 01:14:14.152983 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:14:14.152943    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:14:14.739972 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:14.739928    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 31 01:14:24.740135 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:24.740088    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 31 01:14:31.318620 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:31.318537    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"]
Oct 31 01:14:31.319156 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:31.318739    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" podUID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" containerName="istio-proxy" containerID="cri-o://071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb" gracePeriod=30
Oct 31 01:14:31.365157 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:31.365120    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"]
Oct 31 01:14:31.365434 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:31.365386    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerName="istio-proxy" containerID="cri-o://f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2" gracePeriod=30
Oct 31 01:14:31.366680 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:31.366657    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"]
Oct 31 01:14:31.366873 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:31.366853    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" podUID="06d62ef7-c133-4cdd-84de-14d4a0acb945" containerName="istio-proxy" containerID="cri-o://377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125" gracePeriod=30
Oct 31 01:14:31.411954 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:31.411924    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"]
Oct 31 01:14:31.412148 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:31.412130    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" podUID="608e466f-5bce-4ba9-9327-681d81a5f0f3" containerName="istio-proxy" containerID="cri-o://b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab" gracePeriod=30
Oct 31 01:14:33.228791 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:33.228749    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-v6lqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:14:33.229186 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:33.228822    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:14:34.739477 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:34.739439    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 31 01:14:36.274262 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.274229    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-nfb67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:14:36.274634 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.274282    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" podUID="608e466f-5bce-4ba9-9327-681d81a5f0f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:14:36.622355 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.622319    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kzxcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:14:36.622563 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.622379    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" podUID="06d62ef7-c133-4cdd-84de-14d4a0acb945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 31 01:14:36.650321 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.650294    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:14:36.658280 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.658256    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-kube-api-access-pnsxg\" (UniqueName: \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-kube-api-access-pnsxg\") pod \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") "
Oct 31 01:14:36.658415 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.658308    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-certs\") pod \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") "
Oct 31 01:14:36.658415 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.658363    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-socket\") pod \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") "
Oct 31 01:14:36.658534 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.658439    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-envoy\") pod \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") "
Oct 31 01:14:36.658534 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.658509    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-podinfo\") pod \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") "
Oct 31 01:14:36.658654 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.658544    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istiod-ca-cert\") pod \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") "
Oct 31 01:14:36.658654 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.658571    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-data\") pod \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") "
Oct 31 01:14:36.658654 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.658612    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-token\" (UniqueName: \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-token\") pod \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") "
Oct 31 01:14:36.658766 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.658675    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-credential-socket\") pod \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\" (UID: \"f4d08047-d9b7-4d05-a5da-e6b97b03b36e\") "
Oct 31 01:14:36.659427 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.659241    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-credential-socket" pod "f4d08047-d9b7-4d05-a5da-e6b97b03b36e" (UID: "f4d08047-d9b7-4d05-a5da-e6b97b03b36e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.659427 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.659317    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istiod-ca-cert" pod "f4d08047-d9b7-4d05-a5da-e6b97b03b36e" (UID: "f4d08047-d9b7-4d05-a5da-e6b97b03b36e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:36.659605 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.659475    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-certs" pod "f4d08047-d9b7-4d05-a5da-e6b97b03b36e" (UID: "f4d08047-d9b7-4d05-a5da-e6b97b03b36e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.659965 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.659700    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-data" pod "f4d08047-d9b7-4d05-a5da-e6b97b03b36e" (UID: "f4d08047-d9b7-4d05-a5da-e6b97b03b36e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.660329 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.660293    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-socket" pod "f4d08047-d9b7-4d05-a5da-e6b97b03b36e" (UID: "f4d08047-d9b7-4d05-a5da-e6b97b03b36e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.663721 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.663697    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-token" pod "f4d08047-d9b7-4d05-a5da-e6b97b03b36e" (UID: "f4d08047-d9b7-4d05-a5da-e6b97b03b36e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:36.665224 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.665167    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-envoy" pod "f4d08047-d9b7-4d05-a5da-e6b97b03b36e" (UID: "f4d08047-d9b7-4d05-a5da-e6b97b03b36e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.665224 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.665201    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-podinfo" pod "f4d08047-d9b7-4d05-a5da-e6b97b03b36e" (UID: "f4d08047-d9b7-4d05-a5da-e6b97b03b36e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:14:36.667249 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.667224    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-kube-api-access-pnsxg" pod "f4d08047-d9b7-4d05-a5da-e6b97b03b36e" (UID: "f4d08047-d9b7-4d05-a5da-e6b97b03b36e"). InnerVolumeSpecName "kube-api-access-pnsxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:36.679801 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.679776    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:14:36.726052 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.726007    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:14:36.738103 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.738081    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:14:36.759821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.759799    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-data\") pod \"608e466f-5bce-4ba9-9327-681d81a5f0f3\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") "
Oct 31 01:14:36.759941 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.759844    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-credential-socket\") pod \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") "
Oct 31 01:14:36.759941 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.759870    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-certs\") pod \"608e466f-5bce-4ba9-9327-681d81a5f0f3\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") "
Oct 31 01:14:36.759941 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.759901    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-credential-socket\") pod \"608e466f-5bce-4ba9-9327-681d81a5f0f3\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") "
Oct 31 01:14:36.759941 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.759928    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d62ef7-c133-4cdd-84de-14d4a0acb945-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d62ef7-c133-4cdd-84de-14d4a0acb945-istiod-ca-cert\") pod \"06d62ef7-c133-4cdd-84de-14d4a0acb945\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") "
Oct 31 01:14:36.760184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.759959    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-certs\") pod \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") "
Oct 31 01:14:36.760184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.759990    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-data\") pod \"06d62ef7-c133-4cdd-84de-14d4a0acb945\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") "
Oct 31 01:14:36.760184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760028    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-socket\") pod \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") "
Oct 31 01:14:36.760184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760067    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-token\" (UniqueName: \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-token\") pod \"06d62ef7-c133-4cdd-84de-14d4a0acb945\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") "
Oct 31 01:14:36.760184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760090    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-envoy\") pod \"06d62ef7-c133-4cdd-84de-14d4a0acb945\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") "
Oct 31 01:14:36.760184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760121    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-kube-api-access-s9v6g\" (UniqueName: \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-kube-api-access-s9v6g\") pod \"06d62ef7-c133-4cdd-84de-14d4a0acb945\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") "
Oct 31 01:14:36.760184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760127    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-credential-socket" pod "fe47e516-df31-4e4f-8a14-b8c00990c0f6" (UID: "fe47e516-df31-4e4f-8a14-b8c00990c0f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.760184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760130    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-data" pod "608e466f-5bce-4ba9-9327-681d81a5f0f3" (UID: "608e466f-5bce-4ba9-9327-681d81a5f0f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.760184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760171    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-envoy\") pod \"608e466f-5bce-4ba9-9327-681d81a5f0f3\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") "
Oct 31 01:14:36.760184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760178    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-credential-socket" pod "608e466f-5bce-4ba9-9327-681d81a5f0f3" (UID: "608e466f-5bce-4ba9-9327-681d81a5f0f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760259    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-envoy\") pod \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760328    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-socket\") pod \"06d62ef7-c133-4cdd-84de-14d4a0acb945\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760354    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608e466f-5bce-4ba9-9327-681d81a5f0f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e466f-5bce-4ba9-9327-681d81a5f0f3-istiod-ca-cert\") pod \"608e466f-5bce-4ba9-9327-681d81a5f0f3\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760379    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-socket" pod "fe47e516-df31-4e4f-8a14-b8c00990c0f6" (UID: "fe47e516-df31-4e4f-8a14-b8c00990c0f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760385    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-token\" (UniqueName: \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-token\") pod \"608e466f-5bce-4ba9-9327-681d81a5f0f3\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760412    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-socket\") pod \"608e466f-5bce-4ba9-9327-681d81a5f0f3\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760417    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d62ef7-c133-4cdd-84de-14d4a0acb945-istiod-ca-cert" pod "06d62ef7-c133-4cdd-84de-14d4a0acb945" (UID: "06d62ef7-c133-4cdd-84de-14d4a0acb945"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760425    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-certs" pod "fe47e516-df31-4e4f-8a14-b8c00990c0f6" (UID: "fe47e516-df31-4e4f-8a14-b8c00990c0f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760449    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-token\" (UniqueName: \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-token\") pod \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760479    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-podinfo\") pod \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760511    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-credential-socket\") pod \"06d62ef7-c133-4cdd-84de-14d4a0acb945\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760540    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-kube-api-access-82w8b\" (UniqueName: \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-kube-api-access-82w8b\") pod \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760573    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-certs\") pod \"06d62ef7-c133-4cdd-84de-14d4a0acb945\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760591    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-data" pod "06d62ef7-c133-4cdd-84de-14d4a0acb945" (UID: "06d62ef7-c133-4cdd-84de-14d4a0acb945"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760601    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istiod-ca-cert\") pod \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760630    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-podinfo\") pod \"608e466f-5bce-4ba9-9327-681d81a5f0f3\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") "
Oct 31 01:14:36.760664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760664    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-data\") pod \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\" (UID: \"fe47e516-df31-4e4f-8a14-b8c00990c0f6\") "
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760702    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-kube-api-access-gbmd6\" (UniqueName: \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-kube-api-access-gbmd6\") pod \"608e466f-5bce-4ba9-9327-681d81a5f0f3\" (UID: \"608e466f-5bce-4ba9-9327-681d81a5f0f3\") "
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760746    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-podinfo\") pod \"06d62ef7-c133-4cdd-84de-14d4a0acb945\" (UID: \"06d62ef7-c133-4cdd-84de-14d4a0acb945\") "
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.760954    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-certs" pod "608e466f-5bce-4ba9-9327-681d81a5f0f3" (UID: "608e466f-5bce-4ba9-9327-681d81a5f0f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761240    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761259    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761274    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761288    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761301    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761315    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761328    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761329    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istiod-ca-cert" pod "fe47e516-df31-4e4f-8a14-b8c00990c0f6" (UID: "fe47e516-df31-4e4f-8a14-b8c00990c0f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761344    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761360    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761373    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761389    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d62ef7-c133-4cdd-84de-14d4a0acb945-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761402    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761416    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761431    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnsxg\" (UniqueName: \"kubernetes.io/projected/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-kube-api-access-pnsxg\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761446    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761459    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761473    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d08047-d9b7-4d05-a5da-e6b97b03b36e-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.761693 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.761572    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-credential-socket" pod "06d62ef7-c133-4cdd-84de-14d4a0acb945" (UID: "06d62ef7-c133-4cdd-84de-14d4a0acb945"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.762926 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.762071    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-socket" pod "608e466f-5bce-4ba9-9327-681d81a5f0f3" (UID: "608e466f-5bce-4ba9-9327-681d81a5f0f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.762926 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.762273    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-data" pod "fe47e516-df31-4e4f-8a14-b8c00990c0f6" (UID: "fe47e516-df31-4e4f-8a14-b8c00990c0f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.762926 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.762454    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-socket" pod "06d62ef7-c133-4cdd-84de-14d4a0acb945" (UID: "06d62ef7-c133-4cdd-84de-14d4a0acb945"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.763571 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.763503    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608e466f-5bce-4ba9-9327-681d81a5f0f3-istiod-ca-cert" pod "608e466f-5bce-4ba9-9327-681d81a5f0f3" (UID: "608e466f-5bce-4ba9-9327-681d81a5f0f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:36.763965 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.763930    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-certs" pod "06d62ef7-c133-4cdd-84de-14d4a0acb945" (UID: "06d62ef7-c133-4cdd-84de-14d4a0acb945"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.764247 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.764140    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-envoy" pod "608e466f-5bce-4ba9-9327-681d81a5f0f3" (UID: "608e466f-5bce-4ba9-9327-681d81a5f0f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.764827 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.764802    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-podinfo" pod "608e466f-5bce-4ba9-9327-681d81a5f0f3" (UID: "608e466f-5bce-4ba9-9327-681d81a5f0f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:14:36.764827 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.764815    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-token" pod "06d62ef7-c133-4cdd-84de-14d4a0acb945" (UID: "06d62ef7-c133-4cdd-84de-14d4a0acb945"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:36.765010 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.764800    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-kube-api-access-s9v6g" pod "06d62ef7-c133-4cdd-84de-14d4a0acb945" (UID: "06d62ef7-c133-4cdd-84de-14d4a0acb945"). InnerVolumeSpecName "kube-api-access-s9v6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:36.765010 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.764896    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-podinfo" pod "fe47e516-df31-4e4f-8a14-b8c00990c0f6" (UID: "fe47e516-df31-4e4f-8a14-b8c00990c0f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:14:36.765397 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.765336    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-kube-api-access-gbmd6" pod "608e466f-5bce-4ba9-9327-681d81a5f0f3" (UID: "608e466f-5bce-4ba9-9327-681d81a5f0f3"). InnerVolumeSpecName "kube-api-access-gbmd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:36.765656 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.765636    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-token" pod "608e466f-5bce-4ba9-9327-681d81a5f0f3" (UID: "608e466f-5bce-4ba9-9327-681d81a5f0f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:36.765792 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.765771    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-envoy" pod "06d62ef7-c133-4cdd-84de-14d4a0acb945" (UID: "06d62ef7-c133-4cdd-84de-14d4a0acb945"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.765964 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.765941    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-token" pod "fe47e516-df31-4e4f-8a14-b8c00990c0f6" (UID: "fe47e516-df31-4e4f-8a14-b8c00990c0f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:36.766307 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.766290    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-podinfo" pod "06d62ef7-c133-4cdd-84de-14d4a0acb945" (UID: "06d62ef7-c133-4cdd-84de-14d4a0acb945"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:14:36.766611 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.766589    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-envoy" pod "fe47e516-df31-4e4f-8a14-b8c00990c0f6" (UID: "fe47e516-df31-4e4f-8a14-b8c00990c0f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:36.766788 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.766766    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-kube-api-access-82w8b" pod "fe47e516-df31-4e4f-8a14-b8c00990c0f6" (UID: "fe47e516-df31-4e4f-8a14-b8c00990c0f6"). InnerVolumeSpecName "kube-api-access-82w8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:36.862787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862709    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.862787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862736    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.862787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862746    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9v6g\" (UniqueName: \"kubernetes.io/projected/06d62ef7-c133-4cdd-84de-14d4a0acb945-kube-api-access-s9v6g\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.862787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862757    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.862787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862765    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.862787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862773    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.862787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862781    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608e466f-5bce-4ba9-9327-681d81a5f0f3-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.862787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862788    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.863168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862796    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608e466f-5bce-4ba9-9327-681d81a5f0f3-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.863168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862805    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.863168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862813    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.863168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862821    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.863168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862830    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82w8b\" (UniqueName: \"kubernetes.io/projected/fe47e516-df31-4e4f-8a14-b8c00990c0f6-kube-api-access-82w8b\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.863168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862838    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d62ef7-c133-4cdd-84de-14d4a0acb945-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.863168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862846    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.863168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862853    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608e466f-5bce-4ba9-9327-681d81a5f0f3-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.863168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862862    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe47e516-df31-4e4f-8a14-b8c00990c0f6-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.863168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862870    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbmd6\" (UniqueName: \"kubernetes.io/projected/608e466f-5bce-4ba9-9327-681d81a5f0f3-kube-api-access-gbmd6\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:36.863168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:36.862878    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d62ef7-c133-4cdd-84de-14d4a0acb945-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:37.392468 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.392438    4935 generic.go:356] "Generic (PLEG): container finished" podID="608e466f-5bce-4ba9-9327-681d81a5f0f3" containerID="b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab" exitCode=0
Oct 31 01:14:37.392898 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.392490    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"
Oct 31 01:14:37.392898 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.392524    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" event={"ID":"608e466f-5bce-4ba9-9327-681d81a5f0f3","Type":"ContainerDied","Data":"b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab"}
Oct 31 01:14:37.392898 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.392581    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67" event={"ID":"608e466f-5bce-4ba9-9327-681d81a5f0f3","Type":"ContainerDied","Data":"467eb655c16687d3bc66a7561adb24f2d6029112a4d9f02ab5eacad98b193cfb"}
Oct 31 01:14:37.392898 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.392612    4935 scope.go:122] "RemoveContainer" containerID="b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab"
Oct 31 01:14:37.394069 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.394024    4935 generic.go:356] "Generic (PLEG): container finished" podID="06d62ef7-c133-4cdd-84de-14d4a0acb945" containerID="377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125" exitCode=0
Oct 31 01:14:37.394183 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.394067    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" event={"ID":"06d62ef7-c133-4cdd-84de-14d4a0acb945","Type":"ContainerDied","Data":"377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125"}
Oct 31 01:14:37.394183 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.394134    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"
Oct 31 01:14:37.394308 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.394273    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm" event={"ID":"06d62ef7-c133-4cdd-84de-14d4a0acb945","Type":"ContainerDied","Data":"11b80ad6f99182026310d9c83f3cab53641c0dbdb3dc7817b9c4be6771a9d02d"}
Oct 31 01:14:37.395766 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.395739    4935 generic.go:356] "Generic (PLEG): container finished" podID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerID="f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2" exitCode=0
Oct 31 01:14:37.395879 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.395805    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"
Oct 31 01:14:37.395879 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.395817    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" event={"ID":"fe47e516-df31-4e4f-8a14-b8c00990c0f6","Type":"ContainerDied","Data":"f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2"}
Oct 31 01:14:37.395879 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.395854    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx" event={"ID":"fe47e516-df31-4e4f-8a14-b8c00990c0f6","Type":"ContainerDied","Data":"b40c23b596cbd77b50b31e2dbf45b814836a224c92f753c07e321af7a1b0867d"}
Oct 31 01:14:37.399330 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.399305    4935 generic.go:356] "Generic (PLEG): container finished" podID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" containerID="071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb" exitCode=0
Oct 31 01:14:37.399455 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.399330    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" event={"ID":"f4d08047-d9b7-4d05-a5da-e6b97b03b36e","Type":"ContainerDied","Data":"071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb"}
Oct 31 01:14:37.399455 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.399349    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"
Oct 31 01:14:37.399455 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.399377    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" event={"ID":"f4d08047-d9b7-4d05-a5da-e6b97b03b36e","Type":"ContainerDied","Data":"fd1620eff57a8ec2efb379380bac29f16bcb63f0ab8fb7938b480e31701a7bc1"}
Oct 31 01:14:37.407245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.403377    4935 scope.go:122] "RemoveContainer" containerID="b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab"
Oct 31 01:14:37.407245 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:14:37.403823    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab\": container with ID starting with b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab not found: ID does not exist" containerID="b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab"
Oct 31 01:14:37.407245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.403854    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab"} err="failed to get container status \"b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab\": rpc error: code = NotFound desc = could not find container \"b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab\": container with ID starting with b07a12522a1ded4670b90acfa0f657b8a892e3e3e0fa6bdb08e00864cb7ef6ab not found: ID does not exist"
Oct 31 01:14:37.407245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.403875    4935 scope.go:122] "RemoveContainer" containerID="377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125"
Oct 31 01:14:37.414279 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.414260    4935 scope.go:122] "RemoveContainer" containerID="377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125"
Oct 31 01:14:37.414514 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:14:37.414495    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125\": container with ID starting with 377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125 not found: ID does not exist" containerID="377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125"
Oct 31 01:14:37.414577 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.414521    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125"} err="failed to get container status \"377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125\": rpc error: code = NotFound desc = could not find container \"377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125\": container with ID starting with 377c837aba8c4e09894d31a4dad41f92de6d400ace830fd64abb3481acefe125 not found: ID does not exist"
Oct 31 01:14:37.414577 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.414535    4935 scope.go:122] "RemoveContainer" containerID="f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2"
Oct 31 01:14:37.422167 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.422149    4935 scope.go:122] "RemoveContainer" containerID="f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2"
Oct 31 01:14:37.422430 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:14:37.422412    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2\": container with ID starting with f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2 not found: ID does not exist" containerID="f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2"
Oct 31 01:14:37.422475 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.422435    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2"} err="failed to get container status \"f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2\": rpc error: code = NotFound desc = could not find container \"f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2\": container with ID starting with f1aeeb0c79016722d677bf665566e2d2575a690f2abf9f0dab1eb6b6da5497a2 not found: ID does not exist"
Oct 31 01:14:37.422475 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.422451    4935 scope.go:122] "RemoveContainer" containerID="071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb"
Oct 31 01:14:37.430139 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.430120    4935 scope.go:122] "RemoveContainer" containerID="071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb"
Oct 31 01:14:37.430424 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:14:37.430398    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb\": container with ID starting with 071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb not found: ID does not exist" containerID="071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb"
Oct 31 01:14:37.430473 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.430434    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb"} err="failed to get container status \"071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb\": rpc error: code = NotFound desc = could not find container \"071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb\": container with ID starting with 071b4b234bae5e2bca23df064fae3380def89b145b1f916b0644ee02aaf044fb not found: ID does not exist"
Oct 31 01:14:37.552821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.552786    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"]
Oct 31 01:14:37.599725 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.599661    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-v6lqx"]
Oct 31 01:14:37.629043 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.629013    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-b9b65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:14:37.629155 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.629071    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65" podUID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:14:37.666868 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.666847    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"]
Oct 31 01:14:37.674694 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.674673    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-b9b65"]
Oct 31 01:14:37.784256 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.784229    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"]
Oct 31 01:14:37.790574 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.790547    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kzxcm"]
Oct 31 01:14:37.896091 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.896014    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"]
Oct 31 01:14:37.903866 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.903841    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-nfb67"]
Oct 31 01:14:37.943053 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.943019    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d62ef7-c133-4cdd-84de-14d4a0acb945" path="/var/lib/kubelet/pods/06d62ef7-c133-4cdd-84de-14d4a0acb945/volumes"
Oct 31 01:14:37.943424 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.943410    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608e466f-5bce-4ba9-9327-681d81a5f0f3" path="/var/lib/kubelet/pods/608e466f-5bce-4ba9-9327-681d81a5f0f3/volumes"
Oct 31 01:14:37.943772 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.943761    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" path="/var/lib/kubelet/pods/f4d08047-d9b7-4d05-a5da-e6b97b03b36e/volumes"
Oct 31 01:14:37.944454 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:37.944440    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" path="/var/lib/kubelet/pods/fe47e516-df31-4e4f-8a14-b8c00990c0f6/volumes"
Oct 31 01:14:40.716536 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:40.716503    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"]
Oct 31 01:14:40.716974 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:40.716693    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy" containerID="cri-o://45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2" gracePeriod=30
Oct 31 01:14:41.958942 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.958908    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"]
Oct 31 01:14:41.959513 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959476    4935 cpu_manager.go:414] "containerMap: removing container" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="storage-initializer"
Oct 31 01:14:41.959513 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959497    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="storage-initializer"
Oct 31 01:14:41.959513 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959515    4935 cpu_manager.go:414] "containerMap: removing container" podUID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" containerName="istio-proxy"
Oct 31 01:14:41.959688 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959523    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" containerName="istio-proxy"
Oct 31 01:14:41.959688 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959571    4935 cpu_manager.go:414] "containerMap: removing container" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerName="istio-proxy"
Oct 31 01:14:41.959688 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959578    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerName="istio-proxy"
Oct 31 01:14:41.959688 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959598    4935 cpu_manager.go:414] "containerMap: removing container" podUID="608e466f-5bce-4ba9-9327-681d81a5f0f3" containerName="istio-proxy"
Oct 31 01:14:41.959688 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959607    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608e466f-5bce-4ba9-9327-681d81a5f0f3" containerName="istio-proxy"
Oct 31 01:14:41.959688 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959618    4935 cpu_manager.go:414] "containerMap: removing container" podUID="06d62ef7-c133-4cdd-84de-14d4a0acb945" containerName="istio-proxy"
Oct 31 01:14:41.959688 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959624    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d62ef7-c133-4cdd-84de-14d4a0acb945" containerName="istio-proxy"
Oct 31 01:14:41.959688 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959636    4935 cpu_manager.go:414] "containerMap: removing container" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main"
Oct 31 01:14:41.959688 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959642    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main"
Oct 31 01:14:41.959944 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959727    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe47e516-df31-4e4f-8a14-b8c00990c0f6" containerName="istio-proxy"
Oct 31 01:14:41.959944 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959740    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="608e466f-5bce-4ba9-9327-681d81a5f0f3" containerName="istio-proxy"
Oct 31 01:14:41.959944 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959749    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d62ef7-c133-4cdd-84de-14d4a0acb945" containerName="istio-proxy"
Oct 31 01:14:41.959944 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959757    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6011a2-7b15-456e-b0f0-43f8d7cb0d7a" containerName="main"
Oct 31 01:14:41.959944 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.959768    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d08047-d9b7-4d05-a5da-e6b97b03b36e" containerName="istio-proxy"
Oct 31 01:14:41.965143 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:41.965120    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.009600 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.009572    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"]
Oct 31 01:14:42.012457 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.012427    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.012590 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.012560    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.012663 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.012596    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.012663 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.012623    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.012663 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.012657    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhh9k\" (UniqueName: \"kubernetes.io/projected/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kube-api-access-bhh9k\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.012890 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.012726    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32986fda-715f-452e-b1bb-af4fd7d4ed0f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.012890 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.012779    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.012890 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.012806    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.054754 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.054728    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-vbflt\""
Oct 31 01:14:42.054968 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.054729    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Oct 31 01:14:42.055099 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.054743    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:14:42.055099 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.054740    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Oct 31 01:14:42.055099 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.054770    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Oct 31 01:14:42.055099 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.054811    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Oct 31 01:14:42.114202 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.114167    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.114377 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.114223    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.114437 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.114409    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.114493 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.114460    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.114541 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.114494    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhh9k\" (UniqueName: \"kubernetes.io/projected/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kube-api-access-bhh9k\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.114597 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.114546    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32986fda-715f-452e-b1bb-af4fd7d4ed0f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.114597 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.114585    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.114730 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.114705    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.114815 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.114747    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.114815 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.114800    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.115094 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.115074    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.115266 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.115245    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.115315 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.115246    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32986fda-715f-452e-b1bb-af4fd7d4ed0f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.116725 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.116702    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.116974 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.116953    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.175469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.175438    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhh9k\" (UniqueName: \"kubernetes.io/projected/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kube-api-access-bhh9k\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.282899 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.282856    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:42.724303 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:42.724213    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"]
Oct 31 01:14:42.731171 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:14:42.731143    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32986fda_715f_452e_b1bb_af4fd7d4ed0f.slice/crio-b9a947f5190ff3cb1384d17caf2b5259ec88d764244ed806b06d063fac569017 WatchSource:0}: Error finding container b9a947f5190ff3cb1384d17caf2b5259ec88d764244ed806b06d063fac569017: Status 404 returned error can't find the container with id b9a947f5190ff3cb1384d17caf2b5259ec88d764244ed806b06d063fac569017
Oct 31 01:14:43.434417 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:43.434372    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" event={"ID":"32986fda-715f-452e-b1bb-af4fd7d4ed0f","Type":"ContainerStarted","Data":"ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f"}
Oct 31 01:14:43.434904 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:43.434439    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" event={"ID":"32986fda-715f-452e-b1bb-af4fd7d4ed0f","Type":"ContainerStarted","Data":"b9a947f5190ff3cb1384d17caf2b5259ec88d764244ed806b06d063fac569017"}
Oct 31 01:14:44.161380 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:14:44.161343    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:14:44.740086 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:44.740025    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 31 01:14:45.448140 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:45.448026    4935 generic.go:356] "Generic (PLEG): container finished" podID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerID="ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f" exitCode=0
Oct 31 01:14:45.448140 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:45.448100    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" event={"ID":"32986fda-715f-452e-b1bb-af4fd7d4ed0f","Type":"ContainerDied","Data":"ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f"}
Oct 31 01:14:45.977363 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:45.977332    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:14:46.061645 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.061603    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6edf0352-997e-4594-bb6a-78932b363339-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6edf0352-997e-4594-bb6a-78932b363339-istiod-ca-cert\") pod \"6edf0352-997e-4594-bb6a-78932b363339\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") "
Oct 31 01:14:46.061830 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.061671    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-envoy\") pod \"6edf0352-997e-4594-bb6a-78932b363339\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") "
Oct 31 01:14:46.061830 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.061774    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-socket\") pod \"6edf0352-997e-4594-bb6a-78932b363339\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") "
Oct 31 01:14:46.061959 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.061840    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-certs\") pod \"6edf0352-997e-4594-bb6a-78932b363339\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") "
Oct 31 01:14:46.061959 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.061864    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-data\") pod \"6edf0352-997e-4594-bb6a-78932b363339\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") "
Oct 31 01:14:46.061959 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.061922    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-credential-socket\") pod \"6edf0352-997e-4594-bb6a-78932b363339\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") "
Oct 31 01:14:46.062133 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.061962    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-kube-api-access-bqkwx\" (UniqueName: \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-kube-api-access-bqkwx\") pod \"6edf0352-997e-4594-bb6a-78932b363339\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") "
Oct 31 01:14:46.062133 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.062026    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-istio-token\" (UniqueName: \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-istio-token\") pod \"6edf0352-997e-4594-bb6a-78932b363339\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") "
Oct 31 01:14:46.062133 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.062082    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6edf0352-997e-4594-bb6a-78932b363339-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6edf0352-997e-4594-bb6a-78932b363339-istio-podinfo\") pod \"6edf0352-997e-4594-bb6a-78932b363339\" (UID: \"6edf0352-997e-4594-bb6a-78932b363339\") "
Oct 31 01:14:46.062282 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.062130    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-socket" pod "6edf0352-997e-4594-bb6a-78932b363339" (UID: "6edf0352-997e-4594-bb6a-78932b363339"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:46.062282 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.062161    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6edf0352-997e-4594-bb6a-78932b363339-istiod-ca-cert" pod "6edf0352-997e-4594-bb6a-78932b363339" (UID: "6edf0352-997e-4594-bb6a-78932b363339"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:46.062425 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.062404    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6edf0352-997e-4594-bb6a-78932b363339-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:46.062483 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.062430    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:46.062758 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.062690    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-data" pod "6edf0352-997e-4594-bb6a-78932b363339" (UID: "6edf0352-997e-4594-bb6a-78932b363339"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:46.062927 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.062908    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-credential-socket" pod "6edf0352-997e-4594-bb6a-78932b363339" (UID: "6edf0352-997e-4594-bb6a-78932b363339"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:46.063145 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.063119    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-certs" pod "6edf0352-997e-4594-bb6a-78932b363339" (UID: "6edf0352-997e-4594-bb6a-78932b363339"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:46.064379 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.064356    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6edf0352-997e-4594-bb6a-78932b363339-istio-podinfo" pod "6edf0352-997e-4594-bb6a-78932b363339" (UID: "6edf0352-997e-4594-bb6a-78932b363339"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:14:46.064649 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.064623    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-envoy" pod "6edf0352-997e-4594-bb6a-78932b363339" (UID: "6edf0352-997e-4594-bb6a-78932b363339"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:14:46.064815 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.064797    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-kube-api-access-bqkwx" pod "6edf0352-997e-4594-bb6a-78932b363339" (UID: "6edf0352-997e-4594-bb6a-78932b363339"). InnerVolumeSpecName "kube-api-access-bqkwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:46.064952 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.064935    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-istio-token" pod "6edf0352-997e-4594-bb6a-78932b363339" (UID: "6edf0352-997e-4594-bb6a-78932b363339"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:46.166734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.166579    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:46.166734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.166625    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:46.166734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.166643    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:46.166734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.166658    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6edf0352-997e-4594-bb6a-78932b363339-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:46.166734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.166673    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqkwx\" (UniqueName: \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-kube-api-access-bqkwx\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:46.166734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.166689    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6edf0352-997e-4594-bb6a-78932b363339-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:46.166734 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.166702    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6edf0352-997e-4594-bb6a-78932b363339-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:14:46.455919 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.455825    4935 generic.go:356] "Generic (PLEG): container finished" podID="6edf0352-997e-4594-bb6a-78932b363339" containerID="45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2" exitCode=0
Oct 31 01:14:46.455919 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.455907    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"
Oct 31 01:14:46.456174 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.455916    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" event={"ID":"6edf0352-997e-4594-bb6a-78932b363339","Type":"ContainerDied","Data":"45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2"}
Oct 31 01:14:46.456174 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.455977    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc" event={"ID":"6edf0352-997e-4594-bb6a-78932b363339","Type":"ContainerDied","Data":"b146705c8a4d223ebc8225f4337afc6e63516cb0a5e152f03215217a651e3dae"}
Oct 31 01:14:46.456174 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.456009    4935 scope.go:122] "RemoveContainer" containerID="45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2"
Oct 31 01:14:46.458258 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.458234    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" event={"ID":"32986fda-715f-452e-b1bb-af4fd7d4ed0f","Type":"ContainerStarted","Data":"3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66"}
Oct 31 01:14:46.467383 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.467365    4935 scope.go:122] "RemoveContainer" containerID="d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384"
Oct 31 01:14:46.477203 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.477186    4935 scope.go:122] "RemoveContainer" containerID="45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2"
Oct 31 01:14:46.477510 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:14:46.477491    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2\": container with ID starting with 45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2 not found: ID does not exist" containerID="45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2"
Oct 31 01:14:46.477578 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.477518    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2"} err="failed to get container status \"45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2\": rpc error: code = NotFound desc = could not find container \"45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2\": container with ID starting with 45ea9d1ffd211d2846104c8b70db7be138abcd0bc968ccc2633624ed1238a4f2 not found: ID does not exist"
Oct 31 01:14:46.477578 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.477535    4935 scope.go:122] "RemoveContainer" containerID="d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384"
Oct 31 01:14:46.477804 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:14:46.477787    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384\": container with ID starting with d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384 not found: ID does not exist" containerID="d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384"
Oct 31 01:14:46.477852 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.477813    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384"} err="failed to get container status \"d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384\": rpc error: code = NotFound desc = could not find container \"d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384\": container with ID starting with d7e034404a42cf4283ab0e4b1ca7ff2d842b057cd0dd16f097d81d13d3102384 not found: ID does not exist"
Oct 31 01:14:46.617759 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.617712    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"]
Oct 31 01:14:46.663557 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.663525    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-hljqc"]
Oct 31 01:14:46.743557 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:46.743496    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podStartSLOduration=3.535211131 podStartE2EDuration="5.74348168s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.208270549s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:46.741842701 +0000 UTC m=+2758.924372019" watchObservedRunningTime="2026-10-31 01:14:46.74348168 +0000 UTC m=+2758.926010973"
Oct 31 01:14:47.945496 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:47.945460    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6edf0352-997e-4594-bb6a-78932b363339" path="/var/lib/kubelet/pods/6edf0352-997e-4594-bb6a-78932b363339/volumes"
Oct 31 01:14:52.284002 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:52.283959    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:52.284421 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:52.284063    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:14:52.285971 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:52.285940    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 01:14:54.740103 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:14:54.740059    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 31 01:15:02.284069 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:02.284015    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 01:15:04.739516 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:04.739475    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 31 01:15:12.284054 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:12.283988    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 01:15:14.152115 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:15:14.152073    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:15:14.749542 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:14.749504    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:15:14.756962 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:14.756932    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:15:22.283278 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:22.283238    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 01:15:23.041901 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:23.041866    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"]
Oct 31 01:15:32.283858 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:32.283810    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 01:15:38.065617 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.065579    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" containerID="cri-o://0a8a4a9aa2d041beb90fac24917a0fe5e25f6114049a33addd3280b0eec2e14b" gracePeriod=45
Oct 31 01:15:38.744842 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.744806    4935 generic.go:356] "Generic (PLEG): container finished" podID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerID="0a8a4a9aa2d041beb90fac24917a0fe5e25f6114049a33addd3280b0eec2e14b" exitCode=0
Oct 31 01:15:38.744998 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.744898    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" event={"ID":"0dcd9130-8332-4efd-b5a0-2212f59ec374","Type":"ContainerDied","Data":"0a8a4a9aa2d041beb90fac24917a0fe5e25f6114049a33addd3280b0eec2e14b"}
Oct 31 01:15:38.819212 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.819184    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:15:38.941967 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.941903    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-model-cache\") pod \"0dcd9130-8332-4efd-b5a0-2212f59ec374\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") "
Oct 31 01:15:38.941967 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.941933    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-home\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-home\") pod \"0dcd9130-8332-4efd-b5a0-2212f59ec374\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") "
Oct 31 01:15:38.941967 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.941951    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-tmp-dir\") pod \"0dcd9130-8332-4efd-b5a0-2212f59ec374\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") "
Oct 31 01:15:38.942256 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.941980    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dcd9130-8332-4efd-b5a0-2212f59ec374-kube-api-access-98kvj\" (UniqueName: \"kubernetes.io/projected/0dcd9130-8332-4efd-b5a0-2212f59ec374-kube-api-access-98kvj\") pod \"0dcd9130-8332-4efd-b5a0-2212f59ec374\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") "
Oct 31 01:15:38.942256 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.942022    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcd9130-8332-4efd-b5a0-2212f59ec374-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcd9130-8332-4efd-b5a0-2212f59ec374-tls-certs\") pod \"0dcd9130-8332-4efd-b5a0-2212f59ec374\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") "
Oct 31 01:15:38.942256 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.942077    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcd9130-8332-4efd-b5a0-2212f59ec374-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dcd9130-8332-4efd-b5a0-2212f59ec374-cabundle-cert\") pod \"0dcd9130-8332-4efd-b5a0-2212f59ec374\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") "
Oct 31 01:15:38.942256 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.942116    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-kserve-provision-location\") pod \"0dcd9130-8332-4efd-b5a0-2212f59ec374\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") "
Oct 31 01:15:38.942256 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.942208    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-dshm\") pod \"0dcd9130-8332-4efd-b5a0-2212f59ec374\" (UID: \"0dcd9130-8332-4efd-b5a0-2212f59ec374\") "
Oct 31 01:15:38.942508 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.942261    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-model-cache" pod "0dcd9130-8332-4efd-b5a0-2212f59ec374" (UID: "0dcd9130-8332-4efd-b5a0-2212f59ec374"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:38.942508 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.942466    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:38.942816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.942792    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-home" pod "0dcd9130-8332-4efd-b5a0-2212f59ec374" (UID: "0dcd9130-8332-4efd-b5a0-2212f59ec374"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:38.943005 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.942822    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcd9130-8332-4efd-b5a0-2212f59ec374-cabundle-cert" pod "0dcd9130-8332-4efd-b5a0-2212f59ec374" (UID: "0dcd9130-8332-4efd-b5a0-2212f59ec374"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:38.944634 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.944600    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcd9130-8332-4efd-b5a0-2212f59ec374-kube-api-access-98kvj" pod "0dcd9130-8332-4efd-b5a0-2212f59ec374" (UID: "0dcd9130-8332-4efd-b5a0-2212f59ec374"). InnerVolumeSpecName "kube-api-access-98kvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:38.944634 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.944612    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-dshm" pod "0dcd9130-8332-4efd-b5a0-2212f59ec374" (UID: "0dcd9130-8332-4efd-b5a0-2212f59ec374"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:38.945164 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.945140    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcd9130-8332-4efd-b5a0-2212f59ec374-tls-certs" pod "0dcd9130-8332-4efd-b5a0-2212f59ec374" (UID: "0dcd9130-8332-4efd-b5a0-2212f59ec374"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:38.953752 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.953718    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-tmp-dir" pod "0dcd9130-8332-4efd-b5a0-2212f59ec374" (UID: "0dcd9130-8332-4efd-b5a0-2212f59ec374"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:38.973474 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:38.973442    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-kserve-provision-location" pod "0dcd9130-8332-4efd-b5a0-2212f59ec374" (UID: "0dcd9130-8332-4efd-b5a0-2212f59ec374"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:39.043624 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.043591    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:39.043624 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.043625    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:39.043799 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.043643    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98kvj\" (UniqueName: \"kubernetes.io/projected/0dcd9130-8332-4efd-b5a0-2212f59ec374-kube-api-access-98kvj\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:39.043799 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.043658    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcd9130-8332-4efd-b5a0-2212f59ec374-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:39.043799 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.043672    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dcd9130-8332-4efd-b5a0-2212f59ec374-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:39.043799 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.043687    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:39.043799 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.043700    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcd9130-8332-4efd-b5a0-2212f59ec374-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:39.750161 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.750121    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:15:39.751773 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.751744    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q" event={"ID":"0dcd9130-8332-4efd-b5a0-2212f59ec374","Type":"ContainerDied","Data":"b2273cae772aa56a97ce6ae569e5f237df0a5d72d5c9871eb94b5aaaa83df11d"}
Oct 31 01:15:39.751883 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.751787    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"
Oct 31 01:15:39.751883 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.751809    4935 scope.go:122] "RemoveContainer" containerID="0a8a4a9aa2d041beb90fac24917a0fe5e25f6114049a33addd3280b0eec2e14b"
Oct 31 01:15:39.761172 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.761155    4935 scope.go:122] "RemoveContainer" containerID="7e924e5bed4d6402ce65056ab612dc74dbbef7b428d293f83a9f7c8d21d2c35c"
Oct 31 01:15:39.961412 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:39.961386    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"]
Oct 31 01:15:40.008113 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:40.008027    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-ssp6q"]
Oct 31 01:15:41.943187 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:41.943154    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" path="/var/lib/kubelet/pods/0dcd9130-8332-4efd-b5a0-2212f59ec374/volumes"
Oct 31 01:15:42.283790 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:42.283753    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 01:15:44.151403 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:15:44.151364    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:15:52.284292 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:15:52.284247    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 01:16:02.283376 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:02.283327    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 01:16:12.283385 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:12.283345    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 01:16:14.156413 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:16:14.156376    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:16:20.131338 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131305    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"]
Oct 31 01:16:20.131749 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131645    4935 cpu_manager.go:414] "containerMap: removing container" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="storage-initializer"
Oct 31 01:16:20.131749 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131656    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="storage-initializer"
Oct 31 01:16:20.131749 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131668    4935 cpu_manager.go:414] "containerMap: removing container" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy"
Oct 31 01:16:20.131749 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131674    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy"
Oct 31 01:16:20.131749 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131682    4935 cpu_manager.go:414] "containerMap: removing container" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main"
Oct 31 01:16:20.131749 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131686    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main"
Oct 31 01:16:20.131749 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131699    4935 cpu_manager.go:414] "containerMap: removing container" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy"
Oct 31 01:16:20.131749 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131704    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy"
Oct 31 01:16:20.131982 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131775    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy"
Oct 31 01:16:20.131982 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131782    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcd9130-8332-4efd-b5a0-2212f59ec374" containerName="main"
Oct 31 01:16:20.131982 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.131791    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="6edf0352-997e-4594-bb6a-78932b363339" containerName="istio-proxy"
Oct 31 01:16:20.135960 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.135945    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.181873 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.181837    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-home\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.181873 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.181873    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-dshm\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.182100 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.181981    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe37331-c8ab-4558-ba96-0ff7a2708f77-cabundle-cert\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.182100 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.182056    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tmp-dir\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.182100 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.182080    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.182280 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.182128    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-model-cache\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.182280 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.182149    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tls-certs\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.182280 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.182177    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksq44\" (UniqueName: \"kubernetes.io/projected/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kube-api-access-ksq44\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.187026 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.186999    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"]
Oct 31 01:16:20.232362 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.232340    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Oct 31 01:16:20.232362 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.232338    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:16:20.232564 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.232339    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Oct 31 01:16:20.232564 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.232340    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-4wc7q\""
Oct 31 01:16:20.232564 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.232425    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Oct 31 01:16:20.232564 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.232460    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Oct 31 01:16:20.282910 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.282884    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tls-certs\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283018 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.282913    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksq44\" (UniqueName: \"kubernetes.io/projected/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kube-api-access-ksq44\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283018 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.282939    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-home\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283154 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.283067    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-dshm\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283154 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.283144    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe37331-c8ab-4558-ba96-0ff7a2708f77-cabundle-cert\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283272 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.283177    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tmp-dir\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283272 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.283207    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283272 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.283253    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-model-cache\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283448 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.283355    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-home\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283640 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.283565    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283640 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.283627    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tmp-dir\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283781 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.283643    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-model-cache\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.283932 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.283910    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe37331-c8ab-4558-ba96-0ff7a2708f77-cabundle-cert\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.285688 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.285665    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tls-certs\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.285811 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.285795    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-dshm\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.342940 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.342914    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksq44\" (UniqueName: \"kubernetes.io/projected/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kube-api-access-ksq44\") pod \"lora-single-adapter-test-kserve-865d4b5694-8qcml\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.454522 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.454429    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:20.890479 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.890433    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"]
Oct 31 01:16:20.897049 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:16:20.897006    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe37331_c8ab_4558_ba96_0ff7a2708f77.slice/crio-9f010b6d24f352b87989ed3241e08780cf45a93b8dd82e351999a68e66734bd6 WatchSource:0}: Error finding container 9f010b6d24f352b87989ed3241e08780cf45a93b8dd82e351999a68e66734bd6: Status 404 returned error can't find the container with id 9f010b6d24f352b87989ed3241e08780cf45a93b8dd82e351999a68e66734bd6
Oct 31 01:16:20.947257 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:20.947231    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" event={"ID":"dbe37331-c8ab-4558-ba96-0ff7a2708f77","Type":"ContainerStarted","Data":"9f010b6d24f352b87989ed3241e08780cf45a93b8dd82e351999a68e66734bd6"}
Oct 31 01:16:21.955215 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:21.955166    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" event={"ID":"dbe37331-c8ab-4558-ba96-0ff7a2708f77","Type":"ContainerStarted","Data":"613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341"}
Oct 31 01:16:22.288692 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:22.288639    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 31 01:16:24.972412 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:24.972369    4935 generic.go:356] "Generic (PLEG): container finished" podID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerID="613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341" exitCode=0
Oct 31 01:16:24.972803 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:24.972432    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" event={"ID":"dbe37331-c8ab-4558-ba96-0ff7a2708f77","Type":"ContainerDied","Data":"613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341"}
Oct 31 01:16:25.980684 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:25.980638    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" event={"ID":"dbe37331-c8ab-4558-ba96-0ff7a2708f77","Type":"ContainerStarted","Data":"5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6"}
Oct 31 01:16:26.194640 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:26.194572    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" podStartSLOduration=2.738774642 podStartE2EDuration="6.194553004s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.455778362s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:26.146001478 +0000 UTC m=+2858.328530774" watchObservedRunningTime="2026-10-31 01:16:26.194553004 +0000 UTC m=+2858.377082298"
Oct 31 01:16:26.455498 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:26.455463    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:30.455432 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:30.455397    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:16:30.456911 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:30.456880    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 31 01:16:32.293370 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:32.293332    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:16:32.301029 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:32.301003    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:16:40.455478 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:40.455428    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 31 01:16:44.152619 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:16:44.152582    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:16:50.454936 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:50.454888    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 31 01:16:55.959969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:16:55.959939    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"]
Oct 31 01:17:00.454784 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:00.454742    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 31 01:17:10.455311 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:10.455271    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 31 01:17:10.982871 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:10.982835    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main" containerID="cri-o://3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66" gracePeriod=45
Oct 31 01:17:11.642047 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.642015    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:17:11.719762 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.719733    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tls-certs\" (UniqueName: \"kubernetes.io/secret/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tls-certs\") pod \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") "
Oct 31 01:17:11.719887 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.719773    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kserve-provision-location\") pod \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") "
Oct 31 01:17:11.719887 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.719791    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-model-cache\") pod \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") "
Oct 31 01:17:11.719887 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.719815    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kube-api-access-bhh9k\" (UniqueName: \"kubernetes.io/projected/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kube-api-access-bhh9k\") pod \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") "
Oct 31 01:17:11.720070 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.719959    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tmp-dir\") pod \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") "
Oct 31 01:17:11.720070 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.720016    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-dshm\") pod \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") "
Oct 31 01:17:11.720184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.720072    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-model-cache" pod "32986fda-715f-452e-b1bb-af4fd7d4ed0f" (UID: "32986fda-715f-452e-b1bb-af4fd7d4ed0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:11.720184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.720102    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32986fda-715f-452e-b1bb-af4fd7d4ed0f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32986fda-715f-452e-b1bb-af4fd7d4ed0f-cabundle-cert\") pod \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") "
Oct 31 01:17:11.720184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.720140    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-home\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-home\") pod \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\" (UID: \"32986fda-715f-452e-b1bb-af4fd7d4ed0f\") "
Oct 31 01:17:11.721228 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.720418    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.721228 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.721176    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-home" pod "32986fda-715f-452e-b1bb-af4fd7d4ed0f" (UID: "32986fda-715f-452e-b1bb-af4fd7d4ed0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:11.721397 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.721352    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32986fda-715f-452e-b1bb-af4fd7d4ed0f-cabundle-cert" pod "32986fda-715f-452e-b1bb-af4fd7d4ed0f" (UID: "32986fda-715f-452e-b1bb-af4fd7d4ed0f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:17:11.722491 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.722462    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-dshm" pod "32986fda-715f-452e-b1bb-af4fd7d4ed0f" (UID: "32986fda-715f-452e-b1bb-af4fd7d4ed0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:11.722625 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.722513    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tls-certs" pod "32986fda-715f-452e-b1bb-af4fd7d4ed0f" (UID: "32986fda-715f-452e-b1bb-af4fd7d4ed0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:17:11.722703 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.722682    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kube-api-access-bhh9k" pod "32986fda-715f-452e-b1bb-af4fd7d4ed0f" (UID: "32986fda-715f-452e-b1bb-af4fd7d4ed0f"). InnerVolumeSpecName "kube-api-access-bhh9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:17:11.739698 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.739662    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tmp-dir" pod "32986fda-715f-452e-b1bb-af4fd7d4ed0f" (UID: "32986fda-715f-452e-b1bb-af4fd7d4ed0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:11.746959 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.746937    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kserve-provision-location" pod "32986fda-715f-452e-b1bb-af4fd7d4ed0f" (UID: "32986fda-715f-452e-b1bb-af4fd7d4ed0f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:11.821459 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.821435    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32986fda-715f-452e-b1bb-af4fd7d4ed0f-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.821459 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.821458    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.821596 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.821471    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.821596 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.821484    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.821596 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.821497    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhh9k\" (UniqueName: \"kubernetes.io/projected/32986fda-715f-452e-b1bb-af4fd7d4ed0f-kube-api-access-bhh9k\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.821596 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.821510    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.821596 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:11.821522    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32986fda-715f-452e-b1bb-af4fd7d4ed0f-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:12.197467 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.197437    4935 generic.go:356] "Generic (PLEG): container finished" podID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerID="3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66" exitCode=0
Oct 31 01:17:12.197644 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.197486    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"
Oct 31 01:17:12.197644 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.197516    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" event={"ID":"32986fda-715f-452e-b1bb-af4fd7d4ed0f","Type":"ContainerDied","Data":"3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66"}
Oct 31 01:17:12.197644 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.197585    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq" event={"ID":"32986fda-715f-452e-b1bb-af4fd7d4ed0f","Type":"ContainerDied","Data":"b9a947f5190ff3cb1384d17caf2b5259ec88d764244ed806b06d063fac569017"}
Oct 31 01:17:12.197644 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.197618    4935 scope.go:122] "RemoveContainer" containerID="3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66"
Oct 31 01:17:12.208309 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.208279    4935 scope.go:122] "RemoveContainer" containerID="ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f"
Oct 31 01:17:12.216404 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.216388    4935 scope.go:122] "RemoveContainer" containerID="3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66"
Oct 31 01:17:12.216692 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:17:12.216671    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66\": container with ID starting with 3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66 not found: ID does not exist" containerID="3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66"
Oct 31 01:17:12.216735 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.216702    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66"} err="failed to get container status \"3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66\": rpc error: code = NotFound desc = could not find container \"3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66\": container with ID starting with 3e892212257b64aac7bbca7707b87a803ac16f2a6ae16f8bad935cf09a492c66 not found: ID does not exist"
Oct 31 01:17:12.216735 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.216718    4935 scope.go:122] "RemoveContainer" containerID="ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f"
Oct 31 01:17:12.216981 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:17:12.216960    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f\": container with ID starting with ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f not found: ID does not exist" containerID="ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f"
Oct 31 01:17:12.217066 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.216984    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f"} err="failed to get container status \"ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f\": rpc error: code = NotFound desc = could not find container \"ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f\": container with ID starting with ea0bd66b8b2c69f22bf71fd26c205af974c42642205409642a5148594a62814f not found: ID does not exist"
Oct 31 01:17:12.357661 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.357619    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"]
Oct 31 01:17:12.404422 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:12.404387    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-85xgxrq"]
Oct 31 01:17:13.943723 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:13.943683    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" path="/var/lib/kubelet/pods/32986fda-715f-452e-b1bb-af4fd7d4ed0f/volumes"
Oct 31 01:17:14.155983 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:17:14.155946    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:17:20.455433 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:20.455389    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 31 01:17:30.455597 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:30.455504    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 31 01:17:40.465103 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:40.465075    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:17:40.472467 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:40.472447    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:17:44.152123 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:17:44.152088    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:17:51.249259 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.249222    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"]
Oct 31 01:17:51.249771 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.249749    4935 cpu_manager.go:414] "containerMap: removing container" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="storage-initializer"
Oct 31 01:17:51.249846 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.249774    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="storage-initializer"
Oct 31 01:17:51.249846 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.249807    4935 cpu_manager.go:414] "containerMap: removing container" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main"
Oct 31 01:17:51.249846 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.249816    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main"
Oct 31 01:17:51.249991 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.249923    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="32986fda-715f-452e-b1bb-af4fd7d4ed0f" containerName="main"
Oct 31 01:17:51.254435 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.254411    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.292358 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.292331    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"]
Oct 31 01:17:51.313060 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.313014    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Oct 31 01:17:51.313200 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.313173    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Oct 31 01:17:51.313200 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.313093    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:17:51.313364 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.313069    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Oct 31 01:17:51.313364 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.313225    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-wxft4\""
Oct 31 01:17:51.313364 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.313274    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Oct 31 01:17:51.342709 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.342686    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.342790 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.342721    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a32426-fb51-484e-b586-4bd78b73f46c-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.342790 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.342741    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.342909 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.342804    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a32426-fb51-484e-b586-4bd78b73f46c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.342909 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.342881    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tqxr\" (UniqueName: \"kubernetes.io/projected/34a32426-fb51-484e-b586-4bd78b73f46c-kube-api-access-8tqxr\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.342998 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.342913    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.342998 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.342949    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.342998 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.342979    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.443674 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.443639    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a32426-fb51-484e-b586-4bd78b73f46c-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.443827 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.443681    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.443827 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.443709    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a32426-fb51-484e-b586-4bd78b73f46c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.443930 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.443840    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tqxr\" (UniqueName: \"kubernetes.io/projected/34a32426-fb51-484e-b586-4bd78b73f46c-kube-api-access-8tqxr\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.443930 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.443881    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.443930 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.443904    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.444102 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.443930    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.444102 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.444004    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.444217 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.444194    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.444260 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.444230    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.444303 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.444275    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.444412 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.444396    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.444498 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.444477    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a32426-fb51-484e-b586-4bd78b73f46c-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.446485 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.446465    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.446723 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.446707    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a32426-fb51-484e-b586-4bd78b73f46c-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.503779 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.503720    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tqxr\" (UniqueName: \"kubernetes.io/projected/34a32426-fb51-484e-b586-4bd78b73f46c-kube-api-access-8tqxr\") pod \"custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:51.571807 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:51.571776    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:17:52.000345 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:52.000317    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"]
Oct 31 01:17:52.006720 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:17:52.006694    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a32426_fb51_484e_b586_4bd78b73f46c.slice/crio-2444e72b61106792e6d7a514ecec42bb5309125263766451e6db036a53991848 WatchSource:0}: Error finding container 2444e72b61106792e6d7a514ecec42bb5309125263766451e6db036a53991848: Status 404 returned error can't find the container with id 2444e72b61106792e6d7a514ecec42bb5309125263766451e6db036a53991848
Oct 31 01:17:52.383309 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:52.383220    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" event={"ID":"34a32426-fb51-484e-b586-4bd78b73f46c","Type":"ContainerStarted","Data":"078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac"}
Oct 31 01:17:52.383309 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:52.383285    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" event={"ID":"34a32426-fb51-484e-b586-4bd78b73f46c","Type":"ContainerStarted","Data":"2444e72b61106792e6d7a514ecec42bb5309125263766451e6db036a53991848"}
Oct 31 01:17:54.393668 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:54.393633    4935 generic.go:356] "Generic (PLEG): container finished" podID="34a32426-fb51-484e-b586-4bd78b73f46c" containerID="078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac" exitCode=0
Oct 31 01:17:54.394106 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:54.393705    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" event={"ID":"34a32426-fb51-484e-b586-4bd78b73f46c","Type":"ContainerDied","Data":"078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac"}
Oct 31 01:17:55.401380 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:55.401337    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" event={"ID":"34a32426-fb51-484e-b586-4bd78b73f46c","Type":"ContainerStarted","Data":"9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c"}
Oct 31 01:17:55.614063 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:17:55.613996    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podStartSLOduration=2.389118323 podStartE2EDuration="4.613982366s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.224864043s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:55.56620708 +0000 UTC m=+2947.748736377" watchObservedRunningTime="2026-10-31 01:17:55.613982366 +0000 UTC m=+2947.796511660"
Oct 31 01:18:01.572098 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:01.572061    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:18:01.572592 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:01.572139    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:18:01.573766 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:01.573732    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 01:18:07.324330 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:07.324300    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-865d4b5694-8qcml_dbe37331-c8ab-4558-ba96-0ff7a2708f77/storage-initializer/0.log"
Oct 31 01:18:07.446360 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:07.446333    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-865d4b5694-8qcml_dbe37331-c8ab-4558-ba96-0ff7a2708f77/main/0.log"
Oct 31 01:18:11.572359 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:11.572307    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 01:18:13.551446 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:13.551409    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"]
Oct 31 01:18:14.152401 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:18:14.152361    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:18:21.572719 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:21.572683    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 01:18:28.574057 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:28.573986    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="main" containerID="cri-o://5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6" gracePeriod=45
Oct 31 01:18:29.232545 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.232514    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:18:29.380281 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380201    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-home\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-home\") pod \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") "
Oct 31 01:18:29.380281 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380238    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe37331-c8ab-4558-ba96-0ff7a2708f77-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe37331-c8ab-4558-ba96-0ff7a2708f77-cabundle-cert\") pod \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") "
Oct 31 01:18:29.380281 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380260    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tmp-dir\") pod \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") "
Oct 31 01:18:29.380569 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380301    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kube-api-access-ksq44\" (UniqueName: \"kubernetes.io/projected/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kube-api-access-ksq44\") pod \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") "
Oct 31 01:18:29.380569 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380380    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-model-cache\") pod \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") "
Oct 31 01:18:29.380569 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380426    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kserve-provision-location\") pod \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") "
Oct 31 01:18:29.380569 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380465    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-dshm\") pod \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") "
Oct 31 01:18:29.380569 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380498    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tls-certs\") pod \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\" (UID: \"dbe37331-c8ab-4558-ba96-0ff7a2708f77\") "
Oct 31 01:18:29.380821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380588    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe37331-c8ab-4558-ba96-0ff7a2708f77-cabundle-cert" pod "dbe37331-c8ab-4558-ba96-0ff7a2708f77" (UID: "dbe37331-c8ab-4558-ba96-0ff7a2708f77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:18:29.380821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380613    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-home" pod "dbe37331-c8ab-4558-ba96-0ff7a2708f77" (UID: "dbe37331-c8ab-4558-ba96-0ff7a2708f77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:29.380821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380643    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-model-cache" pod "dbe37331-c8ab-4558-ba96-0ff7a2708f77" (UID: "dbe37331-c8ab-4558-ba96-0ff7a2708f77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:29.380821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380815    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:18:29.381007 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380846    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe37331-c8ab-4558-ba96-0ff7a2708f77-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:18:29.381007 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.380864    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:18:29.383108 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.383076    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kube-api-access-ksq44" pod "dbe37331-c8ab-4558-ba96-0ff7a2708f77" (UID: "dbe37331-c8ab-4558-ba96-0ff7a2708f77"). InnerVolumeSpecName "kube-api-access-ksq44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:29.383108 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.383073    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-dshm" pod "dbe37331-c8ab-4558-ba96-0ff7a2708f77" (UID: "dbe37331-c8ab-4558-ba96-0ff7a2708f77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:29.383302 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.383127    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tls-certs" pod "dbe37331-c8ab-4558-ba96-0ff7a2708f77" (UID: "dbe37331-c8ab-4558-ba96-0ff7a2708f77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:18:29.391698 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.391674    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tmp-dir" pod "dbe37331-c8ab-4558-ba96-0ff7a2708f77" (UID: "dbe37331-c8ab-4558-ba96-0ff7a2708f77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:29.399835 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.399816    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kserve-provision-location" pod "dbe37331-c8ab-4558-ba96-0ff7a2708f77" (UID: "dbe37331-c8ab-4558-ba96-0ff7a2708f77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:18:29.482048 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.482018    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:18:29.482155 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.482054    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:18:29.482155 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.482066    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:18:29.482155 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.482094    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe37331-c8ab-4558-ba96-0ff7a2708f77-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:18:29.482155 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.482103    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksq44\" (UniqueName: \"kubernetes.io/projected/dbe37331-c8ab-4558-ba96-0ff7a2708f77-kube-api-access-ksq44\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:18:29.577261 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.577227    4935 generic.go:356] "Generic (PLEG): container finished" podID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerID="5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6" exitCode=0
Oct 31 01:18:29.577674 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.577284    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"
Oct 31 01:18:29.577674 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.577312    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" event={"ID":"dbe37331-c8ab-4558-ba96-0ff7a2708f77","Type":"ContainerDied","Data":"5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6"}
Oct 31 01:18:29.577674 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.577415    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml" event={"ID":"dbe37331-c8ab-4558-ba96-0ff7a2708f77","Type":"ContainerDied","Data":"9f010b6d24f352b87989ed3241e08780cf45a93b8dd82e351999a68e66734bd6"}
Oct 31 01:18:29.577674 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.577448    4935 scope.go:122] "RemoveContainer" containerID="5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6"
Oct 31 01:18:29.586931 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.586910    4935 scope.go:122] "RemoveContainer" containerID="613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341"
Oct 31 01:18:29.595613 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.595594    4935 scope.go:122] "RemoveContainer" containerID="5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6"
Oct 31 01:18:29.595854 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:18:29.595835    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6\": container with ID starting with 5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6 not found: ID does not exist" containerID="5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6"
Oct 31 01:18:29.595913 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.595862    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6"} err="failed to get container status \"5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6\": rpc error: code = NotFound desc = could not find container \"5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6\": container with ID starting with 5cfc061dad061bfaeaa6880d28745b85aea90eb989fe8291438af3d512f1d6f6 not found: ID does not exist"
Oct 31 01:18:29.595913 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.595878    4935 scope.go:122] "RemoveContainer" containerID="613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341"
Oct 31 01:18:29.596138 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:18:29.596121    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341\": container with ID starting with 613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341 not found: ID does not exist" containerID="613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341"
Oct 31 01:18:29.596192 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.596143    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341"} err="failed to get container status \"613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341\": rpc error: code = NotFound desc = could not find container \"613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341\": container with ID starting with 613638b024482cf66f5eb5d5c4799d01b478272bc54c8be21ff4990bc2068341 not found: ID does not exist"
Oct 31 01:18:29.737575 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.737546    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"]
Oct 31 01:18:29.743172 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.743153    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-865d4b5694-8qcml"]
Oct 31 01:18:29.942975 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:29.942943    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" path="/var/lib/kubelet/pods/dbe37331-c8ab-4558-ba96-0ff7a2708f77/volumes"
Oct 31 01:18:31.572758 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:31.572718    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 01:18:41.573117 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:41.573065    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 01:18:44.151772 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:18:44.151734    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:18:47.208471 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.208436    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"]
Oct 31 01:18:47.208833 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.208811    4935 cpu_manager.go:414] "containerMap: removing container" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="storage-initializer"
Oct 31 01:18:47.208833 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.208823    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="storage-initializer"
Oct 31 01:18:47.208920 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.208849    4935 cpu_manager.go:414] "containerMap: removing container" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="main"
Oct 31 01:18:47.208920 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.208854    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="main"
Oct 31 01:18:47.208984 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.208935    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe37331-c8ab-4558-ba96-0ff7a2708f77" containerName="main"
Oct 31 01:18:47.212108 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.212091    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.229063 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.229021    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tls-certs\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.229178 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.229111    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.229178 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.229152    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.229178 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.229174    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-model-cache\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.229304 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.229205    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgl8f\" (UniqueName: \"kubernetes.io/projected/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kube-api-access-sgl8f\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.229304 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.229219    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-dshm\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.229383 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.229310    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.229383 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.229333    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-home\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.264105 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.264068    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"]
Oct 31 01:18:47.308382 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.308359    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Oct 31 01:18:47.308382 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.308361    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Oct 31 01:18:47.308584 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.308473    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:18:47.308584 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.308497    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Oct 31 01:18:47.308584 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.308520    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Oct 31 01:18:47.308584 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.308581    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-ppqlt\""
Oct 31 01:18:47.329854 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.329827    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tls-certs\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.329965 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.329879    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.329965 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.329925    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.329965 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.329958    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-model-cache\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.330171 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.330008    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgl8f\" (UniqueName: \"kubernetes.io/projected/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kube-api-access-sgl8f\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.330171 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.330104    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-dshm\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.330171 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.330148    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.330319 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.330183    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-home\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.330628 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.330498    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.330628 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.330529    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.330628 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.330580    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-home\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.330859 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.330823    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-model-cache\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.331150 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.331130    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.332761 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.332740    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-dshm\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.332960 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.332942    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tls-certs\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.390345 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.390309    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgl8f\" (UniqueName: \"kubernetes.io/projected/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kube-api-access-sgl8f\") pod \"lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.527379 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.527340    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:47.959261 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:47.959188    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"]
Oct 31 01:18:47.965426 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:18:47.965403    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a42e65_bae2_4c9e_ad52_0ae1799b93e1.slice/crio-edbb6e0ec4b1e8df2fdb31876650ff0aedbd6e47dd7848684623918bd0c97a40 WatchSource:0}: Error finding container edbb6e0ec4b1e8df2fdb31876650ff0aedbd6e47dd7848684623918bd0c97a40: Status 404 returned error can't find the container with id edbb6e0ec4b1e8df2fdb31876650ff0aedbd6e47dd7848684623918bd0c97a40
Oct 31 01:18:48.015743 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:48.015717    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Oct 31 01:18:48.234954 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:48.234875    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:18:48.255253 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:48.255231    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:18:48.664438 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:48.664403    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" event={"ID":"81a42e65-bae2-4c9e-ad52-0ae1799b93e1","Type":"ContainerStarted","Data":"30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469"}
Oct 31 01:18:48.664642 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:48.664467    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" event={"ID":"81a42e65-bae2-4c9e-ad52-0ae1799b93e1","Type":"ContainerStarted","Data":"edbb6e0ec4b1e8df2fdb31876650ff0aedbd6e47dd7848684623918bd0c97a40"}
Oct 31 01:18:51.573136 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:51.573093    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 01:18:52.686540 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:52.686505    4935 generic.go:356] "Generic (PLEG): container finished" podID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerID="30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469" exitCode=0
Oct 31 01:18:52.686923 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:52.686573    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" event={"ID":"81a42e65-bae2-4c9e-ad52-0ae1799b93e1","Type":"ContainerDied","Data":"30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469"}
Oct 31 01:18:53.693988 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:53.693943    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" event={"ID":"81a42e65-bae2-4c9e-ad52-0ae1799b93e1","Type":"ContainerStarted","Data":"4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1"}
Oct 31 01:18:53.906504 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:53.906441    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" podStartSLOduration=2.921845017 podStartE2EDuration="6.906422013s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.984576996s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:53.859291969 +0000 UTC m=+3006.041821261" watchObservedRunningTime="2026-10-31 01:18:53.906422013 +0000 UTC m=+3006.088951307"
Oct 31 01:18:54.527726 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:54.527693    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:57.527820 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:57.527720    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:18:57.529383 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:18:57.529351    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 31 01:19:01.572470 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:01.572410    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 01:19:07.527801 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:07.527765    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 31 01:19:11.572453 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:11.572406    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 01:19:14.156497 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:19:14.156452    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:19:17.528138 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:17.528093    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 31 01:19:21.572322 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:21.572274    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 01:19:27.528061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:27.528001    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 31 01:19:31.572985 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:31.572938    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 31 01:19:37.528213 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:37.528167    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 31 01:19:41.582917 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:41.582878    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:19:41.590619 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:41.590590    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:19:44.151512 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:19:44.151473    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:19:47.527956 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:47.527915    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 31 01:19:55.325212 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:55.325167    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"]
Oct 31 01:19:57.528019 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:19:57.527980    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 31 01:20:07.537393 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:07.537354    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:20:07.544824 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:07.544801    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:20:10.348134 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:10.348099    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main" containerID="cri-o://9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c" gracePeriod=45
Oct 31 01:20:11.028505 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.028479    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:20:11.067652 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.067625    4935 generic.go:356] "Generic (PLEG): container finished" podID="34a32426-fb51-484e-b586-4bd78b73f46c" containerID="9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c" exitCode=0
Oct 31 01:20:11.067790 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.067668    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" event={"ID":"34a32426-fb51-484e-b586-4bd78b73f46c","Type":"ContainerDied","Data":"9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c"}
Oct 31 01:20:11.067790 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.067705    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2" event={"ID":"34a32426-fb51-484e-b586-4bd78b73f46c","Type":"ContainerDied","Data":"2444e72b61106792e6d7a514ecec42bb5309125263766451e6db036a53991848"}
Oct 31 01:20:11.067790 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.067726    4935 scope.go:122] "RemoveContainer" containerID="9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c"
Oct 31 01:20:11.067790 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.067672    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"
Oct 31 01:20:11.078783 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.078763    4935 scope.go:122] "RemoveContainer" containerID="078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac"
Oct 31 01:20:11.086539 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.086514    4935 scope.go:122] "RemoveContainer" containerID="9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c"
Oct 31 01:20:11.086814 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:20:11.086795    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c\": container with ID starting with 9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c not found: ID does not exist" containerID="9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c"
Oct 31 01:20:11.086865 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.086823    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c"} err="failed to get container status \"9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c\": rpc error: code = NotFound desc = could not find container \"9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c\": container with ID starting with 9540f6b14d3344e3f9fb0c759fca7e7ef18524b23aac6140484611060ec2161c not found: ID does not exist"
Oct 31 01:20:11.086865 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.086839    4935 scope.go:122] "RemoveContainer" containerID="078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac"
Oct 31 01:20:11.087109 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:20:11.087093    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac\": container with ID starting with 078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac not found: ID does not exist" containerID="078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac"
Oct 31 01:20:11.087169 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.087112    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac"} err="failed to get container status \"078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac\": rpc error: code = NotFound desc = could not find container \"078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac\": container with ID starting with 078a5167e5681e0f9f148ce2ea69a8569be643e1baab7a5346670644881813ac not found: ID does not exist"
Oct 31 01:20:11.159852 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.159792    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-home\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-home\") pod \"34a32426-fb51-484e-b586-4bd78b73f46c\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") "
Oct 31 01:20:11.159852 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.159831    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a32426-fb51-484e-b586-4bd78b73f46c-tls-certs\" (UniqueName: \"kubernetes.io/secret/34a32426-fb51-484e-b586-4bd78b73f46c-tls-certs\") pod \"34a32426-fb51-484e-b586-4bd78b73f46c\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") "
Oct 31 01:20:11.160064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.159865    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a32426-fb51-484e-b586-4bd78b73f46c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a32426-fb51-484e-b586-4bd78b73f46c-cabundle-cert\") pod \"34a32426-fb51-484e-b586-4bd78b73f46c\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") "
Oct 31 01:20:11.160064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.159896    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-model-cache\") pod \"34a32426-fb51-484e-b586-4bd78b73f46c\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") "
Oct 31 01:20:11.160064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.159920    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-dshm\") pod \"34a32426-fb51-484e-b586-4bd78b73f46c\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") "
Oct 31 01:20:11.160064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.159937    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-tmp-dir\") pod \"34a32426-fb51-484e-b586-4bd78b73f46c\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") "
Oct 31 01:20:11.160064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.159976    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-kserve-provision-location\") pod \"34a32426-fb51-484e-b586-4bd78b73f46c\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") "
Oct 31 01:20:11.160064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.160007    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a32426-fb51-484e-b586-4bd78b73f46c-kube-api-access-8tqxr\" (UniqueName: \"kubernetes.io/projected/34a32426-fb51-484e-b586-4bd78b73f46c-kube-api-access-8tqxr\") pod \"34a32426-fb51-484e-b586-4bd78b73f46c\" (UID: \"34a32426-fb51-484e-b586-4bd78b73f46c\") "
Oct 31 01:20:11.160362 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.160243    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-model-cache" pod "34a32426-fb51-484e-b586-4bd78b73f46c" (UID: "34a32426-fb51-484e-b586-4bd78b73f46c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:11.160414 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.160380    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.160452 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.160428    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a32426-fb51-484e-b586-4bd78b73f46c-cabundle-cert" pod "34a32426-fb51-484e-b586-4bd78b73f46c" (UID: "34a32426-fb51-484e-b586-4bd78b73f46c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:11.161426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.161402    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-home" pod "34a32426-fb51-484e-b586-4bd78b73f46c" (UID: "34a32426-fb51-484e-b586-4bd78b73f46c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:11.162403 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.162373    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-dshm" pod "34a32426-fb51-484e-b586-4bd78b73f46c" (UID: "34a32426-fb51-484e-b586-4bd78b73f46c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:11.162903 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.162874    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a32426-fb51-484e-b586-4bd78b73f46c-kube-api-access-8tqxr" pod "34a32426-fb51-484e-b586-4bd78b73f46c" (UID: "34a32426-fb51-484e-b586-4bd78b73f46c"). InnerVolumeSpecName "kube-api-access-8tqxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:11.163884 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.163861    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a32426-fb51-484e-b586-4bd78b73f46c-tls-certs" pod "34a32426-fb51-484e-b586-4bd78b73f46c" (UID: "34a32426-fb51-484e-b586-4bd78b73f46c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:11.176575 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.176544    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-tmp-dir" pod "34a32426-fb51-484e-b586-4bd78b73f46c" (UID: "34a32426-fb51-484e-b586-4bd78b73f46c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:11.187519 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.187496    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-kserve-provision-location" pod "34a32426-fb51-484e-b586-4bd78b73f46c" (UID: "34a32426-fb51-484e-b586-4bd78b73f46c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:11.261099 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.261065    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.261099 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.261088    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a32426-fb51-484e-b586-4bd78b73f46c-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.261099 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.261096    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a32426-fb51-484e-b586-4bd78b73f46c-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.261099 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.261106    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.261346 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.261114    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.261346 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.261123    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a32426-fb51-484e-b586-4bd78b73f46c-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.261346 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.261133    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tqxr\" (UniqueName: \"kubernetes.io/projected/34a32426-fb51-484e-b586-4bd78b73f46c-kube-api-access-8tqxr\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.485486 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.485416    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"]
Oct 31 01:20:11.493198 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.493167    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5bb5b4c699-wdsn2"]
Oct 31 01:20:11.943658 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:11.943620    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" path="/var/lib/kubelet/pods/34a32426-fb51-484e-b586-4bd78b73f46c/volumes"
Oct 31 01:20:14.152144 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:20:14.152103    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:20:34.429054 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:34.428962    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5_81a42e65-bae2-4c9e-ad52-0ae1799b93e1/storage-initializer/0.log"
Oct 31 01:20:34.555076 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:34.555029    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5_81a42e65-bae2-4c9e-ad52-0ae1799b93e1/main/0.log"
Oct 31 01:20:38.883514 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:38.883484    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"]
Oct 31 01:20:44.151919 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:20:44.151884    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:20:53.905453 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:53.905418    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main" containerID="cri-o://4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1" gracePeriod=45
Oct 31 01:20:54.555415 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.555391    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:20:54.616864 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.616814    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-cabundle-cert\") pod \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") "
Oct 31 01:20:54.616975 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.616897    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tls-certs\") pod \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") "
Oct 31 01:20:54.616975 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.616928    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tmp-dir\") pod \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") "
Oct 31 01:20:54.616975 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.616945    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kserve-provision-location\") pod \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") "
Oct 31 01:20:54.616975 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.616969    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-home\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-home\") pod \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") "
Oct 31 01:20:54.617372 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.616996    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kube-api-access-sgl8f\" (UniqueName: \"kubernetes.io/projected/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kube-api-access-sgl8f\") pod \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") "
Oct 31 01:20:54.617372 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.617022    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-model-cache\") pod \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") "
Oct 31 01:20:54.617372 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.617090    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-dshm\") pod \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\" (UID: \"81a42e65-bae2-4c9e-ad52-0ae1799b93e1\") "
Oct 31 01:20:54.617372 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.617183    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-cabundle-cert" pod "81a42e65-bae2-4c9e-ad52-0ae1799b93e1" (UID: "81a42e65-bae2-4c9e-ad52-0ae1799b93e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:54.617555 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.617420    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-model-cache" pod "81a42e65-bae2-4c9e-ad52-0ae1799b93e1" (UID: "81a42e65-bae2-4c9e-ad52-0ae1799b93e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:54.617555 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.617460    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-home" pod "81a42e65-bae2-4c9e-ad52-0ae1799b93e1" (UID: "81a42e65-bae2-4c9e-ad52-0ae1799b93e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:54.617555 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.617478    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:54.619496 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.619469    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-dshm" pod "81a42e65-bae2-4c9e-ad52-0ae1799b93e1" (UID: "81a42e65-bae2-4c9e-ad52-0ae1799b93e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:54.619592 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.619557    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kube-api-access-sgl8f" pod "81a42e65-bae2-4c9e-ad52-0ae1799b93e1" (UID: "81a42e65-bae2-4c9e-ad52-0ae1799b93e1"). InnerVolumeSpecName "kube-api-access-sgl8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:54.619653 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.619603    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tls-certs" pod "81a42e65-bae2-4c9e-ad52-0ae1799b93e1" (UID: "81a42e65-bae2-4c9e-ad52-0ae1799b93e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:54.630456 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.630433    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tmp-dir" pod "81a42e65-bae2-4c9e-ad52-0ae1799b93e1" (UID: "81a42e65-bae2-4c9e-ad52-0ae1799b93e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:54.640362 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.640340    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kserve-provision-location" pod "81a42e65-bae2-4c9e-ad52-0ae1799b93e1" (UID: "81a42e65-bae2-4c9e-ad52-0ae1799b93e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:54.718493 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.718473    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:54.718493 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.718491    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:54.718603 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.718502    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:54.718603 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.718511    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:54.718603 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.718520    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgl8f\" (UniqueName: \"kubernetes.io/projected/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-kube-api-access-sgl8f\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:54.718603 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.718529    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:54.718603 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:54.718537    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81a42e65-bae2-4c9e-ad52-0ae1799b93e1-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:55.266226 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.266195    4935 generic.go:356] "Generic (PLEG): container finished" podID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerID="4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1" exitCode=0
Oct 31 01:20:55.266587 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.266260    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"
Oct 31 01:20:55.266587 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.266278    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" event={"ID":"81a42e65-bae2-4c9e-ad52-0ae1799b93e1","Type":"ContainerDied","Data":"4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1"}
Oct 31 01:20:55.266587 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.266362    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" event={"ID":"81a42e65-bae2-4c9e-ad52-0ae1799b93e1","Type":"ContainerDied","Data":"edbb6e0ec4b1e8df2fdb31876650ff0aedbd6e47dd7848684623918bd0c97a40"}
Oct 31 01:20:55.266587 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.266393    4935 scope.go:122] "RemoveContainer" containerID="4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1"
Oct 31 01:20:55.275936 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.275920    4935 scope.go:122] "RemoveContainer" containerID="30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469"
Oct 31 01:20:55.287448 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.287423    4935 scope.go:122] "RemoveContainer" containerID="4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1"
Oct 31 01:20:55.287762 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:20:55.287738    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1\": container with ID starting with 4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1 not found: ID does not exist" containerID="4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1"
Oct 31 01:20:55.287853 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.287772    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1"} err="failed to get container status \"4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1\": rpc error: code = NotFound desc = could not find container \"4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1\": container with ID starting with 4d163ea8c21fae8916dd9f3deba4aa5c900764101c010dde1e07604f10e5e9f1 not found: ID does not exist"
Oct 31 01:20:55.287853 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.287796    4935 scope.go:122] "RemoveContainer" containerID="30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469"
Oct 31 01:20:55.288074 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:20:55.288054    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469\": container with ID starting with 30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469 not found: ID does not exist" containerID="30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469"
Oct 31 01:20:55.288117 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.288080    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469"} err="failed to get container status \"30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469\": rpc error: code = NotFound desc = could not find container \"30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469\": container with ID starting with 30e6a1d3eca7a0d17cbaea8c1a7401a2189d82a071c72a8631356ed71fc95469 not found: ID does not exist"
Oct 31 01:20:55.425870 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.425844    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"]
Oct 31 01:20:55.472523 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.472499    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5"]
Oct 31 01:20:55.538824 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.538770    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55bfff4d55-cjhs5" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded"
Oct 31 01:20:55.838820 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.838767    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"]
Oct 31 01:20:55.839133 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.839120    4935 cpu_manager.go:414] "containerMap: removing container" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main"
Oct 31 01:20:55.839182 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.839134    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main"
Oct 31 01:20:55.839182 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.839158    4935 cpu_manager.go:414] "containerMap: removing container" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="storage-initializer"
Oct 31 01:20:55.839182 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.839163    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="storage-initializer"
Oct 31 01:20:55.839182 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.839172    4935 cpu_manager.go:414] "containerMap: removing container" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main"
Oct 31 01:20:55.839182 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.839177    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main"
Oct 31 01:20:55.839328 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.839188    4935 cpu_manager.go:414] "containerMap: removing container" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="storage-initializer"
Oct 31 01:20:55.839328 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.839193    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="storage-initializer"
Oct 31 01:20:55.839328 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.839249    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a32426-fb51-484e-b586-4bd78b73f46c" containerName="main"
Oct 31 01:20:55.839328 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.839259    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" containerName="main"
Oct 31 01:20:55.843934 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.843919    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:55.895210 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.895188    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Oct 31 01:20:55.895302 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.895267    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-7hqmk\""
Oct 31 01:20:55.895355 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.895343    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:20:55.895416 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.895379    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Oct 31 01:20:55.895416 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.895392    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Oct 31 01:20:55.895416 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.895397    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Oct 31 01:20:55.895514 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.895418    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-xk2zv\""
Oct 31 01:20:55.927934 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.927911    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-home\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:55.928008 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.927946    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:55.928008 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.927971    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tls-certs\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:55.928109 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.928025    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-dshm\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:55.928143 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.928107    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:55.928176 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.928139    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6c29\" (UniqueName: \"kubernetes.io/projected/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kube-api-access-x6c29\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:55.928210 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.928173    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-model-cache\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:55.928241 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.928217    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tmp-dir\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:55.944280 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.944249    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a42e65-bae2-4c9e-ad52-0ae1799b93e1" path="/var/lib/kubelet/pods/81a42e65-bae2-4c9e-ad52-0ae1799b93e1/volumes"
Oct 31 01:20:55.944554 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:55.944538    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"]
Oct 31 01:20:56.029058 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029011    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6c29\" (UniqueName: \"kubernetes.io/projected/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kube-api-access-x6c29\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029169 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029079    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-model-cache\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029169 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029112    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tmp-dir\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029169 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029143    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-home\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029191    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029231    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tls-certs\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029259    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-dshm\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029467 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029369    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029546 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029527    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-model-cache\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029607 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029562    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-home\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029605    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tmp-dir\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029797 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029774    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.029872 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.029855    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.031728 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.031699    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-dshm\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.032002 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.031979    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tls-certs\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.089648 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.089599    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6c29\" (UniqueName: \"kubernetes.io/projected/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kube-api-access-x6c29\") pod \"router-with-refs-pd-test-kserve-67bff887b5-lcpvt\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.159907 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.159881    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:56.588583 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:56.588554    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"]
Oct 31 01:20:56.593840 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:20:56.593808    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32cf02a_22bb_4e91_98ec_7e68f21bdbcc.slice/crio-a309ed01606d75fda0d8f31d3213bead19a6a2835c370dfe7d42c887eeaa27cf WatchSource:0}: Error finding container a309ed01606d75fda0d8f31d3213bead19a6a2835c370dfe7d42c887eeaa27cf: Status 404 returned error can't find the container with id a309ed01606d75fda0d8f31d3213bead19a6a2835c370dfe7d42c887eeaa27cf
Oct 31 01:20:57.278695 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:57.278663    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" event={"ID":"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc","Type":"ContainerStarted","Data":"8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079"}
Oct 31 01:20:57.278857 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:57.278716    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" event={"ID":"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc","Type":"ContainerStarted","Data":"a309ed01606d75fda0d8f31d3213bead19a6a2835c370dfe7d42c887eeaa27cf"}
Oct 31 01:20:57.278857 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:57.278743    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:20:58.284574 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:20:58.284538    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" event={"ID":"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc","Type":"ContainerStarted","Data":"65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66"}
Oct 31 01:20:59.457652 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:20:59.457625    4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32cf02a_22bb_4e91_98ec_7e68f21bdbcc.slice/crio-65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:21:00.296414 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:00.296379    4935 generic.go:356] "Generic (PLEG): container finished" podID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerID="65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66" exitCode=0
Oct 31 01:21:00.296574 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:00.296449    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" event={"ID":"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc","Type":"ContainerDied","Data":"65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66"}
Oct 31 01:21:01.303458 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:01.303426    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" event={"ID":"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc","Type":"ContainerStarted","Data":"99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56"}
Oct 31 01:21:01.517417 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:01.517321    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podStartSLOduration=4.433150942 podStartE2EDuration="6.517303193s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.084152251s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:01.517094216 +0000 UTC m=+3133.699623509" watchObservedRunningTime="2026-10-31 01:21:01.517303193 +0000 UTC m=+3133.699832488"
Oct 31 01:21:02.821654 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:02.821619    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"]
Oct 31 01:21:02.825449 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:02.825423    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:02.888664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:02.888633    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"]
Oct 31 01:21:02.934290 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:02.934262    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-6xjnl\""
Oct 31 01:21:02.934290 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:02.934261    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Oct 31 01:21:02.934468 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:02.934307    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Oct 31 01:21:02.934468 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:02.934323    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 31 01:21:02.992319 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:02.992295    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwhl2\" (UniqueName: \"kubernetes.io/projected/e5de061a-20f1-417c-8e92-041ae68fb70d-kube-api-access-mwhl2\") pod \"e2e-pvc-lora-download-n2ljj\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:03.093118 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:03.093053    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwhl2\" (UniqueName: \"kubernetes.io/projected/e5de061a-20f1-417c-8e92-041ae68fb70d-kube-api-access-mwhl2\") pod \"e2e-pvc-lora-download-n2ljj\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:03.153417 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:03.153395    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwhl2\" (UniqueName: \"kubernetes.io/projected/e5de061a-20f1-417c-8e92-041ae68fb70d-kube-api-access-mwhl2\") pod \"e2e-pvc-lora-download-n2ljj\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:06.160446 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:06.160414    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:21:06.160832 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:06.160481    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:21:06.161588 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:06.161546    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Oct 31 01:21:06.174353 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:06.174331    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:21:08.338015 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:08.337977    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"e2e-pvc-lora-download-n2ljj\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:08.438250 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:08.438217    4935 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"e2e-pvc-lora-download-n2ljj\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:08.438494 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:08.438481    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"e2e-pvc-lora-download-n2ljj\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:08.438666 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:08.438645    4935 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"e2e-pvc-lora-download-n2ljj\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:08.488535 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:08.488503    4935 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"e2e-pvc-lora-download-n2ljj\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") DevicePath \"csi-c91d47011ce6895df2c1cdeaef0d42f5e9a4dc25adb6ff53c03e88c14fa2884a\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:08.787965 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:08.787931    4935 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"e2e-pvc-lora-download-n2ljj\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/1156b9f6da2e7e1f6fc992153dd31aa1f302edf6b2d9e519db60383c52b3d2d9/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:08.848830 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:08.848802    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"e2e-pvc-lora-download-n2ljj\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:09.137138 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:09.137056    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:09.506856 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:09.506821    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"]
Oct 31 01:21:09.512964 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:21:09.512939    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5de061a_20f1_417c_8e92_041ae68fb70d.slice/crio-8dd4eee63d09365df09ed4c47f60fdc657049487bfca4d59ccff4aa29b28d592 WatchSource:0}: Error finding container 8dd4eee63d09365df09ed4c47f60fdc657049487bfca4d59ccff4aa29b28d592: Status 404 returned error can't find the container with id 8dd4eee63d09365df09ed4c47f60fdc657049487bfca4d59ccff4aa29b28d592
Oct 31 01:21:10.350072 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:10.348597    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj" event={"ID":"e5de061a-20f1-417c-8e92-041ae68fb70d","Type":"ContainerStarted","Data":"4de71af938be03a8a9314c8a2f2db80ce7417fec5edcfddb1b130782a0069c97"}
Oct 31 01:21:10.350072 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:10.348650    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj" event={"ID":"e5de061a-20f1-417c-8e92-041ae68fb70d","Type":"ContainerStarted","Data":"8dd4eee63d09365df09ed4c47f60fdc657049487bfca4d59ccff4aa29b28d592"}
Oct 31 01:21:10.461819 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:10.461755    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj" podStartSLOduration=11.461736115 podStartE2EDuration="11.461736115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:10.460524441 +0000 UTC m=+3142.643053735" watchObservedRunningTime="2026-10-31 01:21:10.461736115 +0000 UTC m=+3142.644265411"
Oct 31 01:21:12.361085 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:12.361051    4935 generic.go:356] "Generic (PLEG): container finished" podID="e5de061a-20f1-417c-8e92-041ae68fb70d" containerID="4de71af938be03a8a9314c8a2f2db80ce7417fec5edcfddb1b130782a0069c97" exitCode=0
Oct 31 01:21:12.361085 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:12.361075    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj" event={"ID":"e5de061a-20f1-417c-8e92-041ae68fb70d","Type":"ContainerDied","Data":"4de71af938be03a8a9314c8a2f2db80ce7417fec5edcfddb1b130782a0069c97"}
Oct 31 01:21:13.518895 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:13.518874    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:13.693745 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:13.693676    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"e5de061a-20f1-417c-8e92-041ae68fb70d\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") "
Oct 31 01:21:13.693897 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:13.693784    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5de061a-20f1-417c-8e92-041ae68fb70d-kube-api-access-mwhl2\" (UniqueName: \"kubernetes.io/projected/e5de061a-20f1-417c-8e92-041ae68fb70d-kube-api-access-mwhl2\") pod \"e5de061a-20f1-417c-8e92-041ae68fb70d\" (UID: \"e5de061a-20f1-417c-8e92-041ae68fb70d\") "
Oct 31 01:21:13.696106 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:13.696080    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5de061a-20f1-417c-8e92-041ae68fb70d-kube-api-access-mwhl2" pod "e5de061a-20f1-417c-8e92-041ae68fb70d" (UID: "e5de061a-20f1-417c-8e92-041ae68fb70d"). InnerVolumeSpecName "kube-api-access-mwhl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:13.698009 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:13.697988    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9" pod "e5de061a-20f1-417c-8e92-041ae68fb70d" (UID: "e5de061a-20f1-417c-8e92-041ae68fb70d"). InnerVolumeSpecName "pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 01:21:13.795499 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:13.795473    4935 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" "
Oct 31 01:21:13.795644 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:13.795501    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwhl2\" (UniqueName: \"kubernetes.io/projected/e5de061a-20f1-417c-8e92-041ae68fb70d-kube-api-access-mwhl2\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:21:14.152764 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:21:14.152727    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:21:14.371645 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:14.371614    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"
Oct 31 01:21:14.371811 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:14.371611    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj" event={"ID":"e5de061a-20f1-417c-8e92-041ae68fb70d","Type":"ContainerDied","Data":"8dd4eee63d09365df09ed4c47f60fdc657049487bfca4d59ccff4aa29b28d592"}
Oct 31 01:21:14.371811 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:14.371740    4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dd4eee63d09365df09ed4c47f60fdc657049487bfca4d59ccff4aa29b28d592"
Oct 31 01:21:14.765894 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:14.765870    4935 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9") on node "ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 01:21:14.805552 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:14.805524    4935 reconciler_common.go:299] "Volume detached for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"csi-c91d47011ce6895df2c1cdeaef0d42f5e9a4dc25adb6ff53c03e88c14fa2884a\""
Oct 31 01:21:16.161296 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:16.161244    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Oct 31 01:21:26.160899 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:26.160848    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Oct 31 01:21:32.642212 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.642177    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"]
Oct 31 01:21:32.642595 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.642564    4935 cpu_manager.go:414] "containerMap: removing container" podUID="e5de061a-20f1-417c-8e92-041ae68fb70d" containerName="storage-initializer"
Oct 31 01:21:32.642595 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.642576    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5de061a-20f1-417c-8e92-041ae68fb70d" containerName="storage-initializer"
Oct 31 01:21:32.642682 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.642671    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5de061a-20f1-417c-8e92-041ae68fb70d" containerName="storage-initializer"
Oct 31 01:21:32.646118 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.646098    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.698140 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.698110    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"]
Oct 31 01:21:32.743216 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.743175    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Oct 31 01:21:32.743216 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.743183    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:21:32.743436 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.743234    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Oct 31 01:21:32.743436 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.743303    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-6xjnl\""
Oct 31 01:21:32.743436 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.743329    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 31 01:21:32.743436 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.743372    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Oct 31 01:21:32.745869 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.745847    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-dshm\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.745949 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.745880    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.745949 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.745901    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93bf47b9-6dae-4acc-9d9d-3e287c14028c-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.745949 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.745921    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-home\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.746103 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.745982    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.746103 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.746061    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.746103 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.746091    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.847167 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.847142    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nczg\" (UniqueName: \"kubernetes.io/projected/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kube-api-access-6nczg\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.847331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.847176    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-dshm\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.847331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.847197    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.847331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.847262    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93bf47b9-6dae-4acc-9d9d-3e287c14028c-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.847331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.847292    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-home\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.847331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.847312    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.847556 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.847333    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.847644 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.847616    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.847860 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.847830    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-home\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.847860 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.847851    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.848053 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.847942    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.848167 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.848150    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93bf47b9-6dae-4acc-9d9d-3e287c14028c-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.848410 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.848384    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.849987 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.849967    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-dshm\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.849987 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.849978    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:32.948647 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:32.948585    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nczg\" (UniqueName: \"kubernetes.io/projected/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kube-api-access-6nczg\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:33.009752 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:33.009724    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nczg\" (UniqueName: \"kubernetes.io/projected/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kube-api-access-6nczg\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:36.160850 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:36.160799    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Oct 31 01:21:39.303229 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:39.303193    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:39.403483 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:39.403442    4935 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:39.403742 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:39.403725    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:39.403850 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:39.403835    4935 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:39.453498 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:39.453428    4935 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") DevicePath \"csi-c91d47011ce6895df2c1cdeaef0d42f5e9a4dc25adb6ff53c03e88c14fa2884a\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:39.590880 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:39.590850    4935 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/1156b9f6da2e7e1f6fc992153dd31aa1f302edf6b2d9e519db60383c52b3d2d9/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:39.665506 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:39.665474    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:39.863152 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:39.863109    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:40.351673 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:21:40.351637    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bf47b9_6dae_4acc_9d9d_3e287c14028c.slice/crio-7c9123131b8e2282f46c5412210aa3425cfa17d672c4cc32caa39c171a0b27e1 WatchSource:0}: Error finding container 7c9123131b8e2282f46c5412210aa3425cfa17d672c4cc32caa39c171a0b27e1: Status 404 returned error can't find the container with id 7c9123131b8e2282f46c5412210aa3425cfa17d672c4cc32caa39c171a0b27e1
Oct 31 01:21:40.392331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:40.392299    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"]
Oct 31 01:21:40.507017 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:40.506976    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" event={"ID":"93bf47b9-6dae-4acc-9d9d-3e287c14028c","Type":"ContainerStarted","Data":"f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a"}
Oct 31 01:21:40.507188 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:40.507055    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" event={"ID":"93bf47b9-6dae-4acc-9d9d-3e287c14028c","Type":"ContainerStarted","Data":"7c9123131b8e2282f46c5412210aa3425cfa17d672c4cc32caa39c171a0b27e1"}
Oct 31 01:21:43.523294 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:43.523260    4935 generic.go:356] "Generic (PLEG): container finished" podID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerID="f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a" exitCode=0
Oct 31 01:21:43.523766 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:43.523306    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" event={"ID":"93bf47b9-6dae-4acc-9d9d-3e287c14028c","Type":"ContainerDied","Data":"f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a"}
Oct 31 01:21:44.151810 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:21:44.151773    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:21:44.530910 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:44.530877    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" event={"ID":"93bf47b9-6dae-4acc-9d9d-3e287c14028c","Type":"ContainerStarted","Data":"fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5"}
Oct 31 01:21:44.729572 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:44.729511    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" podStartSLOduration=10.57423404 podStartE2EDuration="12.729496672s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.155262632s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:44.728518021 +0000 UTC m=+3176.911047320" watchObservedRunningTime="2026-10-31 01:21:44.729496672 +0000 UTC m=+3176.912025963"
Oct 31 01:21:44.863268 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:44.863169    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:46.160322 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:46.160278    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Oct 31 01:21:49.863767 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:49.863730    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:21:49.865420 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:49.865385    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 31 01:21:49.873595 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:49.873570    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"]
Oct 31 01:21:49.878755 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:49.878736    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:49.968139 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:49.968105    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"]
Oct 31 01:21:50.012256 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.012233    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:21:50.012409 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.012285    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-q9j8d\""
Oct 31 01:21:50.105739 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.105703    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a308a56d-7aef-45de-b856-0f8ac8c475fd-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.105739 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.105745    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.105957 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.105768    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.105957 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.105840    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.105957 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.105902    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.105957 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.105939    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.106199 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.105966    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.106199 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.106084    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkhzf\" (UniqueName: \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-kube-api-access-mkhzf\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.106199 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.106135    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207050 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.206936    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207050 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.206997    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkhzf\" (UniqueName: \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-kube-api-access-mkhzf\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207284 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.207070    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207284 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.207188    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a308a56d-7aef-45de-b856-0f8ac8c475fd-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207284 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.207218    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207284 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.207256    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207494 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.207284    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207494 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.207319    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207494 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.207354    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207494 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.207389    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207701 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.207654    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.207756 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.207731    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.208057 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.208007    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.208417 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.208396    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a308a56d-7aef-45de-b856-0f8ac8c475fd-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.210329 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.210296    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.210570 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.210552    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.267842 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.267805    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.268214 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.268188    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkhzf\" (UniqueName: \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-kube-api-access-mkhzf\") pod \"openshift-ai-inference-openshift-default-568c495856-qrgs4\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.491844 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.491810    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:50.926798 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.926715    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"]
Oct 31 01:21:50.932606 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:21:50.932578    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda308a56d_7aef_45de_b856_0f8ac8c475fd.slice/crio-8ae551f4ae207e7514bd4345aad7fba05fd772da0ecb2774f59167079d46e5b8 WatchSource:0}: Error finding container 8ae551f4ae207e7514bd4345aad7fba05fd772da0ecb2774f59167079d46e5b8: Status 404 returned error can't find the container with id 8ae551f4ae207e7514bd4345aad7fba05fd772da0ecb2774f59167079d46e5b8
Oct 31 01:21:50.935317 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.935281    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:21:50.935387 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.935349    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:21:50.935432 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:50.935391    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:21:51.580006 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:51.579065    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" event={"ID":"a308a56d-7aef-45de-b856-0f8ac8c475fd","Type":"ContainerStarted","Data":"067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3"}
Oct 31 01:21:51.580006 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:51.579166    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" event={"ID":"a308a56d-7aef-45de-b856-0f8ac8c475fd","Type":"ContainerStarted","Data":"8ae551f4ae207e7514bd4345aad7fba05fd772da0ecb2774f59167079d46e5b8"}
Oct 31 01:21:51.697087 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:51.696994    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" podStartSLOduration=2.696975971 podStartE2EDuration="2.696975971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:51.6959422 +0000 UTC m=+3183.878471494" watchObservedRunningTime="2026-10-31 01:21:51.696975971 +0000 UTC m=+3183.879505266"
Oct 31 01:21:52.492437 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:52.492399    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:52.493988 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:52.493961    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-qrgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:21:52.494142 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:52.494018    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" podUID="a308a56d-7aef-45de-b856-0f8ac8c475fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:21:53.492380 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:53.492331    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-qrgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:21:53.492711 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:53.492387    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" podUID="a308a56d-7aef-45de-b856-0f8ac8c475fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:21:54.492807 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:54.492756    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-qrgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:21:54.493398 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:54.492819    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" podUID="a308a56d-7aef-45de-b856-0f8ac8c475fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:21:55.492740 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:55.492702    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-qrgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:21:55.492969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:55.492754    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" podUID="a308a56d-7aef-45de-b856-0f8ac8c475fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:21:56.163061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:56.162357    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Oct 31 01:21:56.492509 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:56.492474    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-qrgs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:21:56.492719 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:56.492531    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" podUID="a308a56d-7aef-45de-b856-0f8ac8c475fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:21:57.497264 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:57.497175    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:57.497700 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:57.497523    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:57.498321 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:57.498297    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:21:59.863460 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:21:59.863411    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 31 01:22:05.002602 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.002554    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"]
Oct 31 01:22:05.006927 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.006897    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.063645 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.063613    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.063804 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.063664    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.063804 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.063728    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l92j\" (UniqueName: \"kubernetes.io/projected/b36efdef-c7ef-4c3e-a887-05961228c7bb-kube-api-access-8l92j\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.063928 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.063811    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.063928 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.063831    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36efdef-c7ef-4c3e-a887-05961228c7bb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.063928 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.063852    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.063928 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.063924    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.064190 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.063944    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.064190 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.063971    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.164591 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.164552    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.164591 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.164589    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l92j\" (UniqueName: \"kubernetes.io/projected/b36efdef-c7ef-4c3e-a887-05961228c7bb-kube-api-access-8l92j\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.164838 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.164622    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.164838 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.164645    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36efdef-c7ef-4c3e-a887-05961228c7bb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.164838 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.164663    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.164838 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.164711    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.164838 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.164737    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.164838 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.164772    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.164838 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.164821    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.165287 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.164970    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.165287 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.165153    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.165403 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.165296    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.165568 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.165542    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36efdef-c7ef-4c3e-a887-05961228c7bb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.165731 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.165707    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.167191 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.167166    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.167738 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.167713    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.203888 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.203858    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"]
Oct 31 01:22:05.225877 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.225849    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36efdef-c7ef-4c3e-a887-05961228c7bb-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.226015 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.225992    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l92j\" (UniqueName: \"kubernetes.io/projected/b36efdef-c7ef-4c3e-a887-05961228c7bb-kube-api-access-8l92j\") pod \"openshift-ai-inference-openshift-default-568c495856-vgwht\" (UID: \"b36efdef-c7ef-4c3e-a887-05961228c7bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.321069 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.320989    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:05.752000 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.751961    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"]
Oct 31 01:22:05.757065 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:22:05.757016    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36efdef_c7ef_4c3e_a887_05961228c7bb.slice/crio-67e264647b7d620cd6e730d948f1517d920beebcf2caf6ca353e504f142df943 WatchSource:0}: Error finding container 67e264647b7d620cd6e730d948f1517d920beebcf2caf6ca353e504f142df943: Status 404 returned error can't find the container with id 67e264647b7d620cd6e730d948f1517d920beebcf2caf6ca353e504f142df943
Oct 31 01:22:05.759557 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.759511    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:22:05.759639 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.759607    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:22:05.759677 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:05.759654    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:22:06.162677 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:06.162274    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Oct 31 01:22:06.683198 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:06.681241    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht" event={"ID":"b36efdef-c7ef-4c3e-a887-05961228c7bb","Type":"ContainerStarted","Data":"a40fe0c549a13f8b024233d81f82a2e451ea8eb555db79c6ea96aa034c300202"}
Oct 31 01:22:06.683198 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:06.681323    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht" event={"ID":"b36efdef-c7ef-4c3e-a887-05961228c7bb","Type":"ContainerStarted","Data":"67e264647b7d620cd6e730d948f1517d920beebcf2caf6ca353e504f142df943"}
Oct 31 01:22:06.846811 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:06.846759    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht" podStartSLOduration=2.846744996 podStartE2EDuration="2.846744996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:06.845778341 +0000 UTC m=+3199.028307652" watchObservedRunningTime="2026-10-31 01:22:06.846744996 +0000 UTC m=+3199.029274289"
Oct 31 01:22:07.321520 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:07.321477    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:07.323005 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:07.322962    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-vgwht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:22:07.323168 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:07.323085    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht" podUID="b36efdef-c7ef-4c3e-a887-05961228c7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:22:08.322186 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:08.322150    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-vgwht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:22:08.322714 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:08.322201    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht" podUID="b36efdef-c7ef-4c3e-a887-05961228c7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:22:09.321667 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:09.321627    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-vgwht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:22:09.321925 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:09.321684    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht" podUID="b36efdef-c7ef-4c3e-a887-05961228c7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:22:09.867537 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:09.867480    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 31 01:22:10.321805 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:10.321766    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-vgwht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:22:10.322011 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:10.321823    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht" podUID="b36efdef-c7ef-4c3e-a887-05961228c7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:22:11.321973 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:11.321936    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-vgwht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:22:11.322485 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:11.322016    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht" podUID="b36efdef-c7ef-4c3e-a887-05961228c7bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:22:12.325218 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:12.325181    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:12.325716 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:12.325517    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:12.326202 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:12.326181    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-vgwht"
Oct 31 01:22:14.154504 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:22:14.154466    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:22:16.160630 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:16.160580    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Oct 31 01:22:19.863601 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:19.863558    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 31 01:22:20.107441 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.107397    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"]
Oct 31 01:22:20.111072 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.111052    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.211320 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.211286    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"]
Oct 31 01:22:20.215279 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.215254    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.215411 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.215315    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.215411 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.215363    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5bab393-518a-4afb-8bba-d2308ae8dca7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.215411 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.215406    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.215548 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.215450    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.215595 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.215550    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.215641 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.215592    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsm4f\" (UniqueName: \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-kube-api-access-wsm4f\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.215677 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.215643    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.215720 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.215690    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.316895 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.316849    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.316895 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.316895    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5bab393-518a-4afb-8bba-d2308ae8dca7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.317166 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.316922    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.317166 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.316954    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.317166 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.316988    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.317166 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.317016    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsm4f\" (UniqueName: \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-kube-api-access-wsm4f\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.317166 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.317080    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.317166 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.317128    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.317475 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.317183    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.317536 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.317481    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.317589 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.317534    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.317735 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.317702    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.317951 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.317928    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5bab393-518a-4afb-8bba-d2308ae8dca7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.318236 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.317999    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.319885 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.319859    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.319976 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.319955    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.377422 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.377360    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.377530 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.377481    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsm4f\" (UniqueName: \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-kube-api-access-wsm4f\") pod \"openshift-ai-inference-openshift-default-568c495856-mkpp5\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.424544 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.424510    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:20.861895 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.861866    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"]
Oct 31 01:22:20.864600 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:22:20.864563    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bab393_518a_4afb_8bba_d2308ae8dca7.slice/crio-6cf880973e5b7ec3a8c8e5374eb707f9c347f0020a2a4b15c3bd3b0cf629f860 WatchSource:0}: Error finding container 6cf880973e5b7ec3a8c8e5374eb707f9c347f0020a2a4b15c3bd3b0cf629f860: Status 404 returned error can't find the container with id 6cf880973e5b7ec3a8c8e5374eb707f9c347f0020a2a4b15c3bd3b0cf629f860
Oct 31 01:22:20.867450 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.867412    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:22:20.867570 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.867496    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:22:20.867570 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:20.867539    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:22:21.774064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:21.773787    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" event={"ID":"a5bab393-518a-4afb-8bba-d2308ae8dca7","Type":"ContainerStarted","Data":"15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96"}
Oct 31 01:22:21.774064 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:21.773877    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" event={"ID":"a5bab393-518a-4afb-8bba-d2308ae8dca7","Type":"ContainerStarted","Data":"6cf880973e5b7ec3a8c8e5374eb707f9c347f0020a2a4b15c3bd3b0cf629f860"}
Oct 31 01:22:21.940493 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:21.940437    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" podStartSLOduration=1.940421715 podStartE2EDuration="1.940421715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:21.939368132 +0000 UTC m=+3214.121897426" watchObservedRunningTime="2026-10-31 01:22:21.940421715 +0000 UTC m=+3214.122951008"
Oct 31 01:22:22.424937 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:22.424890    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:22.426627 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:22.426592    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-mkpp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:22:22.426796 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:22.426645    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" podUID="a5bab393-518a-4afb-8bba-d2308ae8dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 31 01:22:23.425053 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:23.425003    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-mkpp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:22:23.425568 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:23.425082    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" podUID="a5bab393-518a-4afb-8bba-d2308ae8dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 31 01:22:24.425694 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:24.425654    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-mkpp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:22:24.426227 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:24.425709    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" podUID="a5bab393-518a-4afb-8bba-d2308ae8dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 31 01:22:25.425672 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:25.425625    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-mkpp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:22:25.425883 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:25.425675    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" podUID="a5bab393-518a-4afb-8bba-d2308ae8dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 31 01:22:26.162062 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:26.161467    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Oct 31 01:22:26.425613 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:26.425530    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-mkpp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:22:26.425763 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:26.425639    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" podUID="a5bab393-518a-4afb-8bba-d2308ae8dca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 31 01:22:27.428731 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:27.428693    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:27.429259 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:27.429063    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:27.429733 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:27.429713    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:22:29.864161 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:29.864118    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 31 01:22:36.161278 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:36.161228    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Oct 31 01:22:39.863633 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:39.863581    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 31 01:22:44.160200 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:22:44.160154    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:22:46.160497 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:46.160444    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8001/health\": dial tcp 10.132.0.84:8001: connect: connection refused"
Oct 31 01:22:49.864280 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:49.864233    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 31 01:22:56.172433 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:56.172401    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:22:56.186502 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:56.186478    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:22:59.876330 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:59.876301    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:22:59.883601 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:22:59.883579    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:23:12.808423 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:12.808388    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"]
Oct 31 01:23:14.152781 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:23:14.152747    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:23:27.831245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:27.831155    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main" containerID="cri-o://99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56" gracePeriod=45
Oct 31 01:23:28.574608 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.574562    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="llm-d-routing-sidecar" containerID="cri-o://8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079" gracePeriod=45
Oct 31 01:23:28.831410 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.831378    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:23:28.964553 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.964508    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kube-api-access-x6c29\" (UniqueName: \"kubernetes.io/projected/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kube-api-access-x6c29\") pod \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") "
Oct 31 01:23:28.964732 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.964639    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-home\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-home\") pod \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") "
Oct 31 01:23:28.964732 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.964675    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-cabundle-cert\") pod \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") "
Oct 31 01:23:28.964732 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.964711    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kserve-provision-location\") pod \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") "
Oct 31 01:23:28.964878 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.964746    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-model-cache\") pod \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") "
Oct 31 01:23:28.964878 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.964792    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-dshm\") pod \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") "
Oct 31 01:23:28.964878 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.964817    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tmp-dir\") pod \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") "
Oct 31 01:23:28.965003 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.964876    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tls-certs\" (UniqueName: \"kubernetes.io/secret/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tls-certs\") pod \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\" (UID: \"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc\") "
Oct 31 01:23:28.965164 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.965134    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-home" pod "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" (UID: "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:28.965260 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.965239    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:28.966438 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.965495    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-cabundle-cert" pod "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" (UID: "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:28.966438 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.965549    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-model-cache" pod "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" (UID: "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:28.968139 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.968103    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kube-api-access-x6c29" pod "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" (UID: "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc"). InnerVolumeSpecName "kube-api-access-x6c29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:28.968689 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.968661    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-dshm" pod "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" (UID: "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:28.970320 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.970284    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tls-certs" pod "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" (UID: "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:28.984056 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.984014    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tmp-dir" pod "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" (UID: "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:28.988874 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:28.988841    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kserve-provision-location" pod "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" (UID: "e32cf02a-22bb-4e91-98ec-7e68f21bdbcc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:29.066580 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.066518    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6c29\" (UniqueName: \"kubernetes.io/projected/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kube-api-access-x6c29\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:29.066580 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.066547    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:29.066580 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.066560    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:29.066580 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.066572    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:29.066580 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.066584    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:29.066873 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.066595    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:29.066873 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.066605    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:29.128340 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.128313    4935 generic.go:356] "Generic (PLEG): container finished" podID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerID="99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56" exitCode=0
Oct 31 01:23:29.128340 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.128337    4935 generic.go:356] "Generic (PLEG): container finished" podID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerID="8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079" exitCode=0
Oct 31 01:23:29.128504 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.128369    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"
Oct 31 01:23:29.128504 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.128402    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" event={"ID":"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc","Type":"ContainerDied","Data":"99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56"}
Oct 31 01:23:29.128504 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.128456    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" event={"ID":"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc","Type":"ContainerDied","Data":"8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079"}
Oct 31 01:23:29.128504 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.128474    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt" event={"ID":"e32cf02a-22bb-4e91-98ec-7e68f21bdbcc","Type":"ContainerDied","Data":"a309ed01606d75fda0d8f31d3213bead19a6a2835c370dfe7d42c887eeaa27cf"}
Oct 31 01:23:29.128504 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.128496    4935 scope.go:122] "RemoveContainer" containerID="99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56"
Oct 31 01:23:29.137248 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.137224    4935 scope.go:122] "RemoveContainer" containerID="65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66"
Oct 31 01:23:29.145455 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.145436    4935 scope.go:122] "RemoveContainer" containerID="8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079"
Oct 31 01:23:29.152753 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.152738    4935 scope.go:122] "RemoveContainer" containerID="99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56"
Oct 31 01:23:29.153025 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:23:29.153006    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56\": container with ID starting with 99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56 not found: ID does not exist" containerID="99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56"
Oct 31 01:23:29.153134 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.153057    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56"} err="failed to get container status \"99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56\": rpc error: code = NotFound desc = could not find container \"99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56\": container with ID starting with 99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56 not found: ID does not exist"
Oct 31 01:23:29.153134 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.153081    4935 scope.go:122] "RemoveContainer" containerID="65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66"
Oct 31 01:23:29.153354 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:23:29.153338    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66\": container with ID starting with 65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66 not found: ID does not exist" containerID="65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66"
Oct 31 01:23:29.153401 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.153361    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66"} err="failed to get container status \"65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66\": rpc error: code = NotFound desc = could not find container \"65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66\": container with ID starting with 65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66 not found: ID does not exist"
Oct 31 01:23:29.153401 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.153380    4935 scope.go:122] "RemoveContainer" containerID="8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079"
Oct 31 01:23:29.153617 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:23:29.153599    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079\": container with ID starting with 8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079 not found: ID does not exist" containerID="8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079"
Oct 31 01:23:29.153657 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.153623    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079"} err="failed to get container status \"8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079\": rpc error: code = NotFound desc = could not find container \"8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079\": container with ID starting with 8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079 not found: ID does not exist"
Oct 31 01:23:29.153657 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.153639    4935 scope.go:122] "RemoveContainer" containerID="99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56"
Oct 31 01:23:29.153847 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.153831    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56"} err="failed to get container status \"99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56\": rpc error: code = NotFound desc = could not find container \"99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56\": container with ID starting with 99b507aafde55d614df6a34e275489b106b36a72329019431d0b062308dd6e56 not found: ID does not exist"
Oct 31 01:23:29.153897 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.153848    4935 scope.go:122] "RemoveContainer" containerID="65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66"
Oct 31 01:23:29.154095 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.154077    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66"} err="failed to get container status \"65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66\": rpc error: code = NotFound desc = could not find container \"65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66\": container with ID starting with 65b41cb23573c1e056bfd9024df4d4d71a6de6a42b284cb955c96962ce3eba66 not found: ID does not exist"
Oct 31 01:23:29.154149 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.154096    4935 scope.go:122] "RemoveContainer" containerID="8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079"
Oct 31 01:23:29.154645 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.154628    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079"} err="failed to get container status \"8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079\": rpc error: code = NotFound desc = could not find container \"8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079\": container with ID starting with 8efb3e6a0b25eb46885ec85a03fcf1956ccd75b8ee064db2ac64cba9a4cc5079 not found: ID does not exist"
Oct 31 01:23:29.288746 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.288710    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"]
Oct 31 01:23:29.294769 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.294750    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-67bff887b5-lcpvt"]
Oct 31 01:23:29.677478 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.677447    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt_93bf47b9-6dae-4acc-9d9d-3e287c14028c/storage-initializer/0.log"
Oct 31 01:23:29.800756 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.800734    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt_93bf47b9-6dae-4acc-9d9d-3e287c14028c/main/0.log"
Oct 31 01:23:29.943310 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:29.943260    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" path="/var/lib/kubelet/pods/e32cf02a-22bb-4e91-98ec-7e68f21bdbcc/volumes"
Oct 31 01:23:34.429891 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:34.429858    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"]
Oct 31 01:23:44.150971 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:23:44.150938    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:23:48.267794 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:48.267768    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:23:48.285950 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:48.285927    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:23:49.452198 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:49.452148    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main" containerID="cri-o://fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5" gracePeriod=45
Oct 31 01:23:49.877761 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:49.877721    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 31 01:23:50.101101 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.101075    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:23:50.221350 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.221286    4935 generic.go:356] "Generic (PLEG): container finished" podID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerID="fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5" exitCode=0
Oct 31 01:23:50.221350 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.221340    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" event={"ID":"93bf47b9-6dae-4acc-9d9d-3e287c14028c","Type":"ContainerDied","Data":"fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5"}
Oct 31 01:23:50.221497 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.221349    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"
Oct 31 01:23:50.221497 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.221401    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt" event={"ID":"93bf47b9-6dae-4acc-9d9d-3e287c14028c","Type":"ContainerDied","Data":"7c9123131b8e2282f46c5412210aa3425cfa17d672c4cc32caa39c171a0b27e1"}
Oct 31 01:23:50.221497 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.221423    4935 scope.go:122] "RemoveContainer" containerID="fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5"
Oct 31 01:23:50.228652 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.228624    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kube-api-access-6nczg\" (UniqueName: \"kubernetes.io/projected/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kube-api-access-6nczg\") pod \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") "
Oct 31 01:23:50.228765 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.228675    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tmp-dir\") pod \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") "
Oct 31 01:23:50.228765 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.228709    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tls-certs\" (UniqueName: \"kubernetes.io/secret/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tls-certs\") pod \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") "
Oct 31 01:23:50.228885 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.228770    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-model-cache\") pod \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") "
Oct 31 01:23:50.228885 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.228800    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kserve-provision-location\") pod \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") "
Oct 31 01:23:50.228885 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.228832    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-home\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-home\") pod \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") "
Oct 31 01:23:50.229050 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.229014    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") pod \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") "
Oct 31 01:23:50.229119 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.229101    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-dshm\") pod \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") "
Oct 31 01:23:50.229179 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.229142    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-model-cache" pod "93bf47b9-6dae-4acc-9d9d-3e287c14028c" (UID: "93bf47b9-6dae-4acc-9d9d-3e287c14028c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:50.229179 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.229169    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93bf47b9-6dae-4acc-9d9d-3e287c14028c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93bf47b9-6dae-4acc-9d9d-3e287c14028c-cabundle-cert\") pod \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\" (UID: \"93bf47b9-6dae-4acc-9d9d-3e287c14028c\") "
Oct 31 01:23:50.229304 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.229281    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-home" pod "93bf47b9-6dae-4acc-9d9d-3e287c14028c" (UID: "93bf47b9-6dae-4acc-9d9d-3e287c14028c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:50.229676 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.229563    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:50.229676 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.229590    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:50.229676 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.229595    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bf47b9-6dae-4acc-9d9d-3e287c14028c-cabundle-cert" pod "93bf47b9-6dae-4acc-9d9d-3e287c14028c" (UID: "93bf47b9-6dae-4acc-9d9d-3e287c14028c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:50.231541 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.231519    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tls-certs" pod "93bf47b9-6dae-4acc-9d9d-3e287c14028c" (UID: "93bf47b9-6dae-4acc-9d9d-3e287c14028c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:50.231671 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.231530    4935 scope.go:122] "RemoveContainer" containerID="f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a"
Oct 31 01:23:50.231758 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.231634    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kube-api-access-6nczg" pod "93bf47b9-6dae-4acc-9d9d-3e287c14028c" (UID: "93bf47b9-6dae-4acc-9d9d-3e287c14028c"). InnerVolumeSpecName "kube-api-access-6nczg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:50.231886 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.231850    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-dshm" pod "93bf47b9-6dae-4acc-9d9d-3e287c14028c" (UID: "93bf47b9-6dae-4acc-9d9d-3e287c14028c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:50.235350 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.235326    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9" pod "93bf47b9-6dae-4acc-9d9d-3e287c14028c" (UID: "93bf47b9-6dae-4acc-9d9d-3e287c14028c"). InnerVolumeSpecName "pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 01:23:50.239284 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.239258    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kserve-provision-location" pod "93bf47b9-6dae-4acc-9d9d-3e287c14028c" (UID: "93bf47b9-6dae-4acc-9d9d-3e287c14028c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:50.245781 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.245747    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tmp-dir" pod "93bf47b9-6dae-4acc-9d9d-3e287c14028c" (UID: "93bf47b9-6dae-4acc-9d9d-3e287c14028c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:50.251375 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.251351    4935 scope.go:122] "RemoveContainer" containerID="fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5"
Oct 31 01:23:50.251631 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:23:50.251612    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5\": container with ID starting with fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5 not found: ID does not exist" containerID="fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5"
Oct 31 01:23:50.251690 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.251641    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5"} err="failed to get container status \"fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5\": rpc error: code = NotFound desc = could not find container \"fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5\": container with ID starting with fc9ee26974622ee626318eb4edd292f4a978ae12175ca3801ed5a741b5d4b5f5 not found: ID does not exist"
Oct 31 01:23:50.251690 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.251661    4935 scope.go:122] "RemoveContainer" containerID="f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a"
Oct 31 01:23:50.251907 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:23:50.251890    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a\": container with ID starting with f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a not found: ID does not exist" containerID="f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a"
Oct 31 01:23:50.251957 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.251914    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a"} err="failed to get container status \"f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a\": rpc error: code = NotFound desc = could not find container \"f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a\": container with ID starting with f95e55d7941bfdcc81140fc757d79ae1ecf82d1072b49ab7cc4bcf2d470cec3a not found: ID does not exist"
Oct 31 01:23:50.330238 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.330220    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:50.330238 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.330239    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93bf47b9-6dae-4acc-9d9d-3e287c14028c-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:50.330379 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.330250    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nczg\" (UniqueName: \"kubernetes.io/projected/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kube-api-access-6nczg\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:50.330379 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.330260    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:50.330379 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.330268    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93bf47b9-6dae-4acc-9d9d-3e287c14028c-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:50.330379 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.330275    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93bf47b9-6dae-4acc-9d9d-3e287c14028c-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:50.330379 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.330303    4935 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" "
Oct 31 01:23:50.340367 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.340351    4935 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9") on node "ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 01:23:50.431515 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.431491    4935 reconciler_common.go:299] "Volume detached for volume \"pvc-ef3886fe-cee9-438e-a37d-2c5c9c909aee\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04a15bbdf439276e9\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"csi-c91d47011ce6895df2c1cdeaef0d42f5e9a4dc25adb6ff53c03e88c14fa2884a\""
Oct 31 01:23:50.639180 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.639153    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"]
Oct 31 01:23:50.645763 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:50.645744    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-bd5ff56db-qjswt"]
Oct 31 01:23:51.942821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:51.942790    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" path="/var/lib/kubelet/pods/93bf47b9-6dae-4acc-9d9d-3e287c14028c/volumes"
Oct 31 01:23:58.444266 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444233    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"]
Oct 31 01:23:58.444632 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444572    4935 cpu_manager.go:414] "containerMap: removing container" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="storage-initializer"
Oct 31 01:23:58.444632 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444582    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="storage-initializer"
Oct 31 01:23:58.444632 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444595    4935 cpu_manager.go:414] "containerMap: removing container" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main"
Oct 31 01:23:58.444632 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444601    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main"
Oct 31 01:23:58.444632 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444610    4935 cpu_manager.go:414] "containerMap: removing container" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="storage-initializer"
Oct 31 01:23:58.444632 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444614    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="storage-initializer"
Oct 31 01:23:58.444632 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444626    4935 cpu_manager.go:414] "containerMap: removing container" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="llm-d-routing-sidecar"
Oct 31 01:23:58.444632 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444630    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="llm-d-routing-sidecar"
Oct 31 01:23:58.444891 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444643    4935 cpu_manager.go:414] "containerMap: removing container" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main"
Oct 31 01:23:58.444891 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444648    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main"
Oct 31 01:23:58.444891 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444700    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="llm-d-routing-sidecar"
Oct 31 01:23:58.444891 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444708    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bf47b9-6dae-4acc-9d9d-3e287c14028c" containerName="main"
Oct 31 01:23:58.444891 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.444718    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32cf02a-22bb-4e91-98ec-7e68f21bdbcc" containerName="main"
Oct 31 01:23:58.449231 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.449211    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:23:58.503545 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.503522    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Oct 31 01:23:58.505178 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.505157    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"]
Oct 31 01:23:58.547072 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.547025    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Oct 31 01:23:58.547072 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.547060    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-j6xmj\""
Oct 31 01:23:58.547221 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.547084    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Oct 31 01:23:58.695883 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.695833    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcbjm\" (UniqueName: \"kubernetes.io/projected/6da78492-43f9-404d-b5cd-bcb78a768a55-kube-api-access-wcbjm\") pod \"e2e-pvc-lora-download-gm269\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:23:58.796420 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.796399    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcbjm\" (UniqueName: \"kubernetes.io/projected/6da78492-43f9-404d-b5cd-bcb78a768a55-kube-api-access-wcbjm\") pod \"e2e-pvc-lora-download-gm269\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:23:58.856553 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:58.856530    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcbjm\" (UniqueName: \"kubernetes.io/projected/6da78492-43f9-404d-b5cd-bcb78a768a55-kube-api-access-wcbjm\") pod \"e2e-pvc-lora-download-gm269\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:23:59.739956 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:59.739933    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"]
Oct 31 01:23:59.745632 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:59.745612    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-n2ljj"]
Oct 31 01:23:59.943865 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:23:59.943840    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5de061a-20f1-417c-8e92-041ae68fb70d" path="/var/lib/kubelet/pods/e5de061a-20f1-417c-8e92-041ae68fb70d/volumes"
Oct 31 01:24:03.628381 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:03.628344    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ec5bdde6-46c0-4edb-9a26-6ff01a632858\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586\") pod \"e2e-pvc-lora-download-gm269\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:24:03.728138 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:03.728109    4935 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-ec5bdde6-46c0-4edb-9a26-6ff01a632858\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586\") pod \"e2e-pvc-lora-download-gm269\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:24:03.729230 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:03.729211    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ec5bdde6-46c0-4edb-9a26-6ff01a632858\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586\") pod \"e2e-pvc-lora-download-gm269\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:24:03.729331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:03.729319    4935 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ec5bdde6-46c0-4edb-9a26-6ff01a632858\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586\") pod \"e2e-pvc-lora-download-gm269\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:24:03.779334 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:03.779313    4935 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ec5bdde6-46c0-4edb-9a26-6ff01a632858\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586\") pod \"e2e-pvc-lora-download-gm269\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") DevicePath \"csi-b9644f4e1be76f05ddce4c13ad5a204ff1588b388488cd6de545be8d830b8625\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:24:04.046587 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:04.046558    4935 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ec5bdde6-46c0-4edb-9a26-6ff01a632858\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586\") pod \"e2e-pvc-lora-download-gm269\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3cd41d0745fdb09edb7f8ffc0d307a1def7432292f2a29e1de776e8cfe5262fd/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:24:04.108238 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:04.108212    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ec5bdde6-46c0-4edb-9a26-6ff01a632858\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586\") pod \"e2e-pvc-lora-download-gm269\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:24:04.160152 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:04.160130    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:24:04.530467 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:04.530438    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"]
Oct 31 01:24:04.537117 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:24:04.537086    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6da78492_43f9_404d_b5cd_bcb78a768a55.slice/crio-5d99cba1056bf981033be7bc40a26a0b57dbffb9381115e4363572078e9daa00 WatchSource:0}: Error finding container 5d99cba1056bf981033be7bc40a26a0b57dbffb9381115e4363572078e9daa00: Status 404 returned error can't find the container with id 5d99cba1056bf981033be7bc40a26a0b57dbffb9381115e4363572078e9daa00
Oct 31 01:24:05.286788 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:05.286755    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269" event={"ID":"6da78492-43f9-404d-b5cd-bcb78a768a55","Type":"ContainerStarted","Data":"90b9c60fcd313f7652ef161b49f88d71b4fdcea4740ce0a4a162c56599836118"}
Oct 31 01:24:05.287330 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:05.286803    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269" event={"ID":"6da78492-43f9-404d-b5cd-bcb78a768a55","Type":"ContainerStarted","Data":"5d99cba1056bf981033be7bc40a26a0b57dbffb9381115e4363572078e9daa00"}
Oct 31 01:24:05.398324 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:05.398258    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269" podStartSLOduration=10.398240008 podStartE2EDuration="10.398240008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:05.398098821 +0000 UTC m=+3317.580628115" watchObservedRunningTime="2026-10-31 01:24:05.398240008 +0000 UTC m=+3317.580769302"
Oct 31 01:24:07.296256 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:07.296225    4935 generic.go:356] "Generic (PLEG): container finished" podID="6da78492-43f9-404d-b5cd-bcb78a768a55" containerID="90b9c60fcd313f7652ef161b49f88d71b4fdcea4740ce0a4a162c56599836118" exitCode=0
Oct 31 01:24:07.296577 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:07.296298    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269" event={"ID":"6da78492-43f9-404d-b5cd-bcb78a768a55","Type":"ContainerDied","Data":"90b9c60fcd313f7652ef161b49f88d71b4fdcea4740ce0a4a162c56599836118"}
Oct 31 01:24:08.445205 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:08.445182    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:24:08.468836 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:08.468817    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586\") pod \"6da78492-43f9-404d-b5cd-bcb78a768a55\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") "
Oct 31 01:24:08.468937 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:08.468876    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da78492-43f9-404d-b5cd-bcb78a768a55-kube-api-access-wcbjm\" (UniqueName: \"kubernetes.io/projected/6da78492-43f9-404d-b5cd-bcb78a768a55-kube-api-access-wcbjm\") pod \"6da78492-43f9-404d-b5cd-bcb78a768a55\" (UID: \"6da78492-43f9-404d-b5cd-bcb78a768a55\") "
Oct 31 01:24:08.471122 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:08.471096    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da78492-43f9-404d-b5cd-bcb78a768a55-kube-api-access-wcbjm" pod "6da78492-43f9-404d-b5cd-bcb78a768a55" (UID: "6da78492-43f9-404d-b5cd-bcb78a768a55"). InnerVolumeSpecName "kube-api-access-wcbjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:08.471497 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:08.471479    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586" pod "6da78492-43f9-404d-b5cd-bcb78a768a55" (UID: "6da78492-43f9-404d-b5cd-bcb78a768a55"). InnerVolumeSpecName "pvc-ec5bdde6-46c0-4edb-9a26-6ff01a632858". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 31 01:24:08.570122 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:08.570066    4935 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ec5bdde6-46c0-4edb-9a26-6ff01a632858\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" "
Oct 31 01:24:08.570122 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:08.570097    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcbjm\" (UniqueName: \"kubernetes.io/projected/6da78492-43f9-404d-b5cd-bcb78a768a55-kube-api-access-wcbjm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:09.306439 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:09.306409    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269" event={"ID":"6da78492-43f9-404d-b5cd-bcb78a768a55","Type":"ContainerDied","Data":"5d99cba1056bf981033be7bc40a26a0b57dbffb9381115e4363572078e9daa00"}
Oct 31 01:24:09.306570 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:09.306445    4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d99cba1056bf981033be7bc40a26a0b57dbffb9381115e4363572078e9daa00"
Oct 31 01:24:09.306570 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:09.306423    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"
Oct 31 01:24:09.541547 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:09.541525    4935 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-ec5bdde6-46c0-4edb-9a26-6ff01a632858" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586") on node "ip-10-0-132-125.us-west-2.compute.internal"
Oct 31 01:24:09.577680 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:09.577652    4935 reconciler_common.go:299] "Volume detached for volume \"pvc-ec5bdde6-46c0-4edb-9a26-6ff01a632858\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1b72cd7aeb1e586\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"csi-b9644f4e1be76f05ddce4c13ad5a204ff1588b388488cd6de545be8d830b8625\""
Oct 31 01:24:14.151541 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:24:14.151508    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:24:14.474029 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.473945    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"]
Oct 31 01:24:14.474436 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.474418    4935 cpu_manager.go:414] "containerMap: removing container" podUID="6da78492-43f9-404d-b5cd-bcb78a768a55" containerName="storage-initializer"
Oct 31 01:24:14.474527 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.474439    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da78492-43f9-404d-b5cd-bcb78a768a55" containerName="storage-initializer"
Oct 31 01:24:14.474582 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.474555    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da78492-43f9-404d-b5cd-bcb78a768a55" containerName="storage-initializer"
Oct 31 01:24:14.477893 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.477874    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.513567 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.513536    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.513664 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.513580    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5w9\" (UniqueName: \"kubernetes.io/projected/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kube-api-access-cz5w9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.513721 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.513699    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.513781 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.513762    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.513891 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.513865    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.513956 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.513908    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.514017 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.513991    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.514104 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.514027    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.528917 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.528894    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Oct 31 01:24:14.574894 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.574870    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-whm4l\""
Oct 31 01:24:14.575004 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.574871    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:24:14.575004 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.574873    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Oct 31 01:24:14.575004 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.574870    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Oct 31 01:24:14.575137 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.575123    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Oct 31 01:24:14.577081 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.577063    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"]
Oct 31 01:24:14.615234 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.615208    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.615327 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.615271    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.615327 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.615296    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.615432 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.615326    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.615432 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.615358    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.615522 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.615504    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.615658 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.615640    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5w9\" (UniqueName: \"kubernetes.io/projected/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kube-api-access-cz5w9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.615753 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.615725    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.615814 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.615787    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.615870 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.615814    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.616049 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.616013    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.616109 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.616091    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.616148 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.616091    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.617793 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.617771    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.617864 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.617814    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.676958 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.676934    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5w9\" (UniqueName: \"kubernetes.io/projected/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kube-api-access-cz5w9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:14.792109 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:14.792083    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:15.226682 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:15.226611    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"]
Oct 31 01:24:15.232341 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:24:15.232319    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f4a337_1d9c_4b31_ac36_0d947b6b1544.slice/crio-11abb99960000582170c5aad668764bdcfacccc0f9242208eb157c8d6bbb3b1b WatchSource:0}: Error finding container 11abb99960000582170c5aad668764bdcfacccc0f9242208eb157c8d6bbb3b1b: Status 404 returned error can't find the container with id 11abb99960000582170c5aad668764bdcfacccc0f9242208eb157c8d6bbb3b1b
Oct 31 01:24:15.341205 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:15.341157    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" event={"ID":"e8f4a337-1d9c-4b31-ac36-0d947b6b1544","Type":"ContainerStarted","Data":"43bb9769f4e684832c712fcdfa90a9c49f4964f9d52186bb6d1e2479a7bc45c4"}
Oct 31 01:24:15.341366 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:15.341267    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" event={"ID":"e8f4a337-1d9c-4b31-ac36-0d947b6b1544","Type":"ContainerStarted","Data":"11abb99960000582170c5aad668764bdcfacccc0f9242208eb157c8d6bbb3b1b"}
Oct 31 01:24:17.352162 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:17.352134    4935 generic.go:356] "Generic (PLEG): container finished" podID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerID="43bb9769f4e684832c712fcdfa90a9c49f4964f9d52186bb6d1e2479a7bc45c4" exitCode=0
Oct 31 01:24:17.352631 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:17.352209    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" event={"ID":"e8f4a337-1d9c-4b31-ac36-0d947b6b1544","Type":"ContainerDied","Data":"43bb9769f4e684832c712fcdfa90a9c49f4964f9d52186bb6d1e2479a7bc45c4"}
Oct 31 01:24:18.358857 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:18.358819    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" event={"ID":"e8f4a337-1d9c-4b31-ac36-0d947b6b1544","Type":"ContainerStarted","Data":"e4ba3ebae971888215017a175a1eef97d29e020546abb15cf4dac5864678f0d7"}
Oct 31 01:24:18.572047 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:18.571986    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podStartSLOduration=2.527960432 podStartE2EDuration="4.571973303s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.044012871s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:18.524674578 +0000 UTC m=+3330.707203872" watchObservedRunningTime="2026-10-31 01:24:18.571973303 +0000 UTC m=+3330.754502596"
Oct 31 01:24:18.792998 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:18.792970    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:24.793100 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:24.793062    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:24:24.794459 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:24.794432    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 31 01:24:34.793254 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:34.793208    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 31 01:24:44.152274 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:24:44.152235    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:24:44.793506 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:44.793464    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 31 01:24:54.793199 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:24:54.793152    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 31 01:25:04.792908 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:25:04.792868    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 31 01:25:14.153263 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:25:14.153227    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:25:14.792887 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:25:14.792838    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 31 01:25:24.792547 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:25:24.792496    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 31 01:25:34.793105 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:25:34.793064    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 31 01:25:44.152353 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:25:44.152310    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:25:44.792537 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:25:44.792497    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 31 01:25:54.792701 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:25:54.792662    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 31 01:26:04.802425 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:04.802391    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:26:04.809814 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:04.809790    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:26:12.314178 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:12.314147    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"]
Oct 31 01:26:14.151530 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:26:14.151486    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:26:27.336903 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:27.336800    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" containerID="cri-o://e4ba3ebae971888215017a175a1eef97d29e020546abb15cf4dac5864678f0d7" gracePeriod=45
Oct 31 01:26:27.803847 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:27.803799    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 31 01:26:27.982636 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:27.982605    4935 generic.go:356] "Generic (PLEG): container finished" podID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerID="e4ba3ebae971888215017a175a1eef97d29e020546abb15cf4dac5864678f0d7" exitCode=0
Oct 31 01:26:27.982775 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:27.982670    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" event={"ID":"e8f4a337-1d9c-4b31-ac36-0d947b6b1544","Type":"ContainerDied","Data":"e4ba3ebae971888215017a175a1eef97d29e020546abb15cf4dac5864678f0d7"}
Oct 31 01:26:28.002706 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.002687    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:26:28.159401 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.159351    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kserve-provision-location\") pod \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") "
Oct 31 01:26:28.159401 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.159380    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-model-cache\") pod \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") "
Oct 31 01:26:28.159529 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.159417    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-dshm\") pod \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") "
Oct 31 01:26:28.159529 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.159480    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-cabundle-cert\") pod \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") "
Oct 31 01:26:28.159529 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.159515    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tls-certs\") pod \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") "
Oct 31 01:26:28.159647 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.159556    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kube-api-access-cz5w9\" (UniqueName: \"kubernetes.io/projected/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kube-api-access-cz5w9\") pod \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") "
Oct 31 01:26:28.159647 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.159588    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tmp-dir\") pod \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") "
Oct 31 01:26:28.159647 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.159628    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-home\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-home\") pod \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\" (UID: \"e8f4a337-1d9c-4b31-ac36-0d947b6b1544\") "
Oct 31 01:26:28.159801 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.159693    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-model-cache" pod "e8f4a337-1d9c-4b31-ac36-0d947b6b1544" (UID: "e8f4a337-1d9c-4b31-ac36-0d947b6b1544"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:28.159941 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.159912    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:28.159941 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.159919    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-cabundle-cert" pod "e8f4a337-1d9c-4b31-ac36-0d947b6b1544" (UID: "e8f4a337-1d9c-4b31-ac36-0d947b6b1544"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:28.160722 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.160692    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-home" pod "e8f4a337-1d9c-4b31-ac36-0d947b6b1544" (UID: "e8f4a337-1d9c-4b31-ac36-0d947b6b1544"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:28.161964 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.161939    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kube-api-access-cz5w9" pod "e8f4a337-1d9c-4b31-ac36-0d947b6b1544" (UID: "e8f4a337-1d9c-4b31-ac36-0d947b6b1544"). InnerVolumeSpecName "kube-api-access-cz5w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:28.162085 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.161973    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tls-certs" pod "e8f4a337-1d9c-4b31-ac36-0d947b6b1544" (UID: "e8f4a337-1d9c-4b31-ac36-0d947b6b1544"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:26:28.162085 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.161991    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-dshm" pod "e8f4a337-1d9c-4b31-ac36-0d947b6b1544" (UID: "e8f4a337-1d9c-4b31-ac36-0d947b6b1544"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:28.182177 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.182155    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tmp-dir" pod "e8f4a337-1d9c-4b31-ac36-0d947b6b1544" (UID: "e8f4a337-1d9c-4b31-ac36-0d947b6b1544"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:28.194171 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.194137    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kserve-provision-location" pod "e8f4a337-1d9c-4b31-ac36-0d947b6b1544" (UID: "e8f4a337-1d9c-4b31-ac36-0d947b6b1544"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:28.261205 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.261183    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:28.261205 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.261204    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:28.261316 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.261213    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:28.261316 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.261221    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:28.261316 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.261230    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz5w9\" (UniqueName: \"kubernetes.io/projected/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-kube-api-access-cz5w9\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:28.261316 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.261260    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:28.261316 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.261269    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f4a337-1d9c-4b31-ac36-0d947b6b1544-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:28.988763 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.988740    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"
Oct 31 01:26:28.989158 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.988762    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx" event={"ID":"e8f4a337-1d9c-4b31-ac36-0d947b6b1544","Type":"ContainerDied","Data":"11abb99960000582170c5aad668764bdcfacccc0f9242208eb157c8d6bbb3b1b"}
Oct 31 01:26:28.989158 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.988819    4935 scope.go:122] "RemoveContainer" containerID="e4ba3ebae971888215017a175a1eef97d29e020546abb15cf4dac5864678f0d7"
Oct 31 01:26:28.997887 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:28.997870    4935 scope.go:122] "RemoveContainer" containerID="43bb9769f4e684832c712fcdfa90a9c49f4964f9d52186bb6d1e2479a7bc45c4"
Oct 31 01:26:29.196826 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:29.196805    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"]
Oct 31 01:26:29.243500 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:29.243452    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-775d759698xt2sx"]
Oct 31 01:26:29.943729 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:29.943692    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" path="/var/lib/kubelet/pods/e8f4a337-1d9c-4b31-ac36-0d947b6b1544/volumes"
Oct 31 01:26:40.344030 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.343995    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 31 01:26:40.344503 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.344386    4935 cpu_manager.go:414] "containerMap: removing container" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="storage-initializer"
Oct 31 01:26:40.344503 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.344399    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="storage-initializer"
Oct 31 01:26:40.344503 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.344436    4935 cpu_manager.go:414] "containerMap: removing container" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main"
Oct 31 01:26:40.344503 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.344444    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main"
Oct 31 01:26:40.344503 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.344502    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f4a337-1d9c-4b31-ac36-0d947b6b1544" containerName="main"
Oct 31 01:26:40.349711 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.349673    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.408000 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.407974    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 31 01:26:40.453426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.453404    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.453560 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.453456    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.453560 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.453529    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.453707 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.453565    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkw8t\" (UniqueName: \"kubernetes.io/projected/6432403d-d204-441e-a1c1-0a27fad6a1c2-kube-api-access-rkw8t\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.453707 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.453641    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6432403d-d204-441e-a1c1-0a27fad6a1c2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.453707 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.453657    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6432403d-d204-441e-a1c1-0a27fad6a1c2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.453842 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.453718    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.453842 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.453765    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.455764 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.455744    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-s5g94\""
Oct 31 01:26:40.455764 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.455752    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Oct 31 01:26:40.455911 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.455779    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-zvgr4\""
Oct 31 01:26:40.455911 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.455843    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Oct 31 01:26:40.458365 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.458347    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:26:40.458495 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.458482    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Oct 31 01:26:40.458588 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.458575    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Oct 31 01:26:40.554377 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554347    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6432403d-d204-441e-a1c1-0a27fad6a1c2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.554518 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554383    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6432403d-d204-441e-a1c1-0a27fad6a1c2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.554518 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554408    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.554518 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554429    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.554518 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554452    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.554518 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554505    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.554737 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554528    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.554737 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554553    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkw8t\" (UniqueName: \"kubernetes.io/projected/6432403d-d204-441e-a1c1-0a27fad6a1c2-kube-api-access-rkw8t\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.554833 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554804    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.554915 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554892    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.554983 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554917    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.554983 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.554925    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.555129 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.555107    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6432403d-d204-441e-a1c1-0a27fad6a1c2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.556887 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.556854    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.557074 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.557055    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6432403d-d204-441e-a1c1-0a27fad6a1c2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.614019 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.613968    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkw8t\" (UniqueName: \"kubernetes.io/projected/6432403d-d204-441e-a1c1-0a27fad6a1c2-kube-api-access-rkw8t\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:40.665119 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:40.665093    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:41.091567 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:41.091536    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 31 01:26:41.098654 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:26:41.098618    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6432403d_d204_441e_a1c1_0a27fad6a1c2.slice/crio-1621817b3c92422ee0c982ab4def096b3289b37d5a49669cee1fb142194acde0 WatchSource:0}: Error finding container 1621817b3c92422ee0c982ab4def096b3289b37d5a49669cee1fb142194acde0: Status 404 returned error can't find the container with id 1621817b3c92422ee0c982ab4def096b3289b37d5a49669cee1fb142194acde0
Oct 31 01:26:42.045772 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:42.045734    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6432403d-d204-441e-a1c1-0a27fad6a1c2","Type":"ContainerStarted","Data":"57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9"}
Oct 31 01:26:42.046184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:42.045788    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6432403d-d204-441e-a1c1-0a27fad6a1c2","Type":"ContainerStarted","Data":"1621817b3c92422ee0c982ab4def096b3289b37d5a49669cee1fb142194acde0"}
Oct 31 01:26:44.072442 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:44.072387    4935 generic.go:356] "Generic (PLEG): container finished" podID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerID="57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9" exitCode=0
Oct 31 01:26:44.073093 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:44.073063    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6432403d-d204-441e-a1c1-0a27fad6a1c2","Type":"ContainerDied","Data":"57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9"}
Oct 31 01:26:44.152267 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:26:44.152237    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:26:45.080260 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:45.080225    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6432403d-d204-441e-a1c1-0a27fad6a1c2","Type":"ContainerStarted","Data":"b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef"}
Oct 31 01:26:45.290603 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:45.290543    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.2513622890000002 podStartE2EDuration="5.29053099s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.039168701s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:45.24322705 +0000 UTC m=+3477.425756345" watchObservedRunningTime="2026-10-31 01:26:45.29053099 +0000 UTC m=+3477.473060284"
Oct 31 01:26:45.665593 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:45.665561    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:50.665228 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:50.665197    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:26:50.666755 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:26:50.666726    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 31 01:27:00.665994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:00.665951    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 31 01:27:10.666313 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:10.666266    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 31 01:27:14.152885 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:27:14.152835    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:27:20.665783 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:20.665742    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 31 01:27:26.845209 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:26.845174    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"]
Oct 31 01:27:26.850239 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:26.850215    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-gm269"]
Oct 31 01:27:27.943684 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:27.943653    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da78492-43f9-404d-b5cd-bcb78a768a55" path="/var/lib/kubelet/pods/6da78492-43f9-404d-b5cd-bcb78a768a55/volumes"
Oct 31 01:27:30.665807 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:30.665761    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 31 01:27:37.540459 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:37.540426    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"]
Oct 31 01:27:37.540879 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:37.540627    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" podUID="a5bab393-518a-4afb-8bba-d2308ae8dca7" containerName="istio-proxy" containerID="cri-o://15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96" gracePeriod=30
Oct 31 01:27:37.587829 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:37.587784    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"]
Oct 31 01:27:37.588083 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:37.588015    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" podUID="a308a56d-7aef-45de-b856-0f8ac8c475fd" containerName="istio-proxy" containerID="cri-o://067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3" gracePeriod=30
Oct 31 01:27:40.665823 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:40.665775    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 31 01:27:42.431425 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.431387    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-mkpp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:27:42.431841 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.431449    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" podUID="a5bab393-518a-4afb-8bba-d2308ae8dca7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:27:42.499770 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.499739    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-qrgs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:27:42.499906 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.499787    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" podUID="a308a56d-7aef-45de-b856-0f8ac8c475fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:27:42.835139 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.835115    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:27:42.862988 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.862962    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:27:42.907812 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.907787    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-credential-socket\") pod \"a308a56d-7aef-45de-b856-0f8ac8c475fd\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") "
Oct 31 01:27:42.907901 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.907821    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-certs\") pod \"a5bab393-518a-4afb-8bba-d2308ae8dca7\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") "
Oct 31 01:27:42.907901 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.907850    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-kube-api-access-mkhzf\" (UniqueName: \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-kube-api-access-mkhzf\") pod \"a308a56d-7aef-45de-b856-0f8ac8c475fd\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") "
Oct 31 01:27:42.907901 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.907881    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-data\") pod \"a308a56d-7aef-45de-b856-0f8ac8c475fd\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") "
Oct 31 01:27:42.908055 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.907916    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-credential-socket\") pod \"a5bab393-518a-4afb-8bba-d2308ae8dca7\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") "
Oct 31 01:27:42.908055 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.907947    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-podinfo\") pod \"a308a56d-7aef-45de-b856-0f8ac8c475fd\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") "
Oct 31 01:27:42.908055 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.907985    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-socket\") pod \"a308a56d-7aef-45de-b856-0f8ac8c475fd\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") "
Oct 31 01:27:42.908055 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908012    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-data\") pod \"a5bab393-518a-4afb-8bba-d2308ae8dca7\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") "
Oct 31 01:27:42.908263 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908061    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-socket\") pod \"a5bab393-518a-4afb-8bba-d2308ae8dca7\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") "
Oct 31 01:27:42.908263 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908077    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-credential-socket" pod "a308a56d-7aef-45de-b856-0f8ac8c475fd" (UID: "a308a56d-7aef-45de-b856-0f8ac8c475fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:27:42.908263 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908121    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-certs\") pod \"a308a56d-7aef-45de-b856-0f8ac8c475fd\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") "
Oct 31 01:27:42.908263 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908130    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-certs" pod "a5bab393-518a-4afb-8bba-d2308ae8dca7" (UID: "a5bab393-518a-4afb-8bba-d2308ae8dca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:27:42.908263 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908161    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-podinfo\") pod \"a5bab393-518a-4afb-8bba-d2308ae8dca7\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") "
Oct 31 01:27:42.908263 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908198    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-token\" (UniqueName: \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-token\") pod \"a308a56d-7aef-45de-b856-0f8ac8c475fd\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") "
Oct 31 01:27:42.908263 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908253    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a308a56d-7aef-45de-b856-0f8ac8c475fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a308a56d-7aef-45de-b856-0f8ac8c475fd-istiod-ca-cert\") pod \"a308a56d-7aef-45de-b856-0f8ac8c475fd\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") "
Oct 31 01:27:42.908641 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908283    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5bab393-518a-4afb-8bba-d2308ae8dca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5bab393-518a-4afb-8bba-d2308ae8dca7-istiod-ca-cert\") pod \"a5bab393-518a-4afb-8bba-d2308ae8dca7\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") "
Oct 31 01:27:42.908641 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908343    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-token\" (UniqueName: \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-token\") pod \"a5bab393-518a-4afb-8bba-d2308ae8dca7\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") "
Oct 31 01:27:42.908641 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908371    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-kube-api-access-wsm4f\" (UniqueName: \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-kube-api-access-wsm4f\") pod \"a5bab393-518a-4afb-8bba-d2308ae8dca7\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") "
Oct 31 01:27:42.908641 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908401    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-envoy\") pod \"a308a56d-7aef-45de-b856-0f8ac8c475fd\" (UID: \"a308a56d-7aef-45de-b856-0f8ac8c475fd\") "
Oct 31 01:27:42.908641 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908433    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-envoy\") pod \"a5bab393-518a-4afb-8bba-d2308ae8dca7\" (UID: \"a5bab393-518a-4afb-8bba-d2308ae8dca7\") "
Oct 31 01:27:42.909079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908226    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-data" pod "a308a56d-7aef-45de-b856-0f8ac8c475fd" (UID: "a308a56d-7aef-45de-b856-0f8ac8c475fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:27:42.909079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908755    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:42.909079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908776    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:42.909079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908791    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:42.909079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908297    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-credential-socket" pod "a5bab393-518a-4afb-8bba-d2308ae8dca7" (UID: "a5bab393-518a-4afb-8bba-d2308ae8dca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:27:42.909079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908313    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-socket" pod "a308a56d-7aef-45de-b856-0f8ac8c475fd" (UID: "a308a56d-7aef-45de-b856-0f8ac8c475fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:27:42.909079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908332    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-socket" pod "a5bab393-518a-4afb-8bba-d2308ae8dca7" (UID: "a5bab393-518a-4afb-8bba-d2308ae8dca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:27:42.909079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908524    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-data" pod "a5bab393-518a-4afb-8bba-d2308ae8dca7" (UID: "a5bab393-518a-4afb-8bba-d2308ae8dca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:27:42.909079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908532    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-certs" pod "a308a56d-7aef-45de-b856-0f8ac8c475fd" (UID: "a308a56d-7aef-45de-b856-0f8ac8c475fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:27:42.909079 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.908658    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a308a56d-7aef-45de-b856-0f8ac8c475fd-istiod-ca-cert" pod "a308a56d-7aef-45de-b856-0f8ac8c475fd" (UID: "a308a56d-7aef-45de-b856-0f8ac8c475fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:27:42.909592 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.909235    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bab393-518a-4afb-8bba-d2308ae8dca7-istiod-ca-cert" pod "a5bab393-518a-4afb-8bba-d2308ae8dca7" (UID: "a5bab393-518a-4afb-8bba-d2308ae8dca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:27:42.911246 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.911115    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-kube-api-access-mkhzf" pod "a308a56d-7aef-45de-b856-0f8ac8c475fd" (UID: "a308a56d-7aef-45de-b856-0f8ac8c475fd"). InnerVolumeSpecName "kube-api-access-mkhzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:27:42.911246 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.911115    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-kube-api-access-wsm4f" pod "a5bab393-518a-4afb-8bba-d2308ae8dca7" (UID: "a5bab393-518a-4afb-8bba-d2308ae8dca7"). InnerVolumeSpecName "kube-api-access-wsm4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:27:42.911246 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.911203    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-podinfo" pod "a308a56d-7aef-45de-b856-0f8ac8c475fd" (UID: "a308a56d-7aef-45de-b856-0f8ac8c475fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:27:42.911434 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.911265    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-token" pod "a5bab393-518a-4afb-8bba-d2308ae8dca7" (UID: "a5bab393-518a-4afb-8bba-d2308ae8dca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:27:42.911680 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.911658    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-envoy" pod "a5bab393-518a-4afb-8bba-d2308ae8dca7" (UID: "a5bab393-518a-4afb-8bba-d2308ae8dca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:27:42.911893 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.911868    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-podinfo" pod "a5bab393-518a-4afb-8bba-d2308ae8dca7" (UID: "a5bab393-518a-4afb-8bba-d2308ae8dca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:27:42.912464 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.912441    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-token" pod "a308a56d-7aef-45de-b856-0f8ac8c475fd" (UID: "a308a56d-7aef-45de-b856-0f8ac8c475fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:27:42.912580 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:42.912565    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-envoy" pod "a308a56d-7aef-45de-b856-0f8ac8c475fd" (UID: "a308a56d-7aef-45de-b856-0f8ac8c475fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:27:43.009645 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009621    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009645 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009647    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009661    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009673    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009687    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009699    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009712    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009724    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a308a56d-7aef-45de-b856-0f8ac8c475fd-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009737    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5bab393-518a-4afb-8bba-d2308ae8dca7-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009748    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009763    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsm4f\" (UniqueName: \"kubernetes.io/projected/a5bab393-518a-4afb-8bba-d2308ae8dca7-kube-api-access-wsm4f\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009775    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a308a56d-7aef-45de-b856-0f8ac8c475fd-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009787    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009798    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkhzf\" (UniqueName: \"kubernetes.io/projected/a308a56d-7aef-45de-b856-0f8ac8c475fd-kube-api-access-mkhzf\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.009855 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.009812    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5bab393-518a-4afb-8bba-d2308ae8dca7-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:27:43.349477 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.349402    4935 generic.go:356] "Generic (PLEG): container finished" podID="a308a56d-7aef-45de-b856-0f8ac8c475fd" containerID="067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3" exitCode=0
Oct 31 01:27:43.349647 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.349513    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"
Oct 31 01:27:43.349647 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.349515    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" event={"ID":"a308a56d-7aef-45de-b856-0f8ac8c475fd","Type":"ContainerDied","Data":"067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3"}
Oct 31 01:27:43.349776 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.349666    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4" event={"ID":"a308a56d-7aef-45de-b856-0f8ac8c475fd","Type":"ContainerDied","Data":"8ae551f4ae207e7514bd4345aad7fba05fd772da0ecb2774f59167079d46e5b8"}
Oct 31 01:27:43.349776 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.349707    4935 scope.go:122] "RemoveContainer" containerID="067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3"
Oct 31 01:27:43.351002 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.350983    4935 generic.go:356] "Generic (PLEG): container finished" podID="a5bab393-518a-4afb-8bba-d2308ae8dca7" containerID="15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96" exitCode=0
Oct 31 01:27:43.351127 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.351012    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" event={"ID":"a5bab393-518a-4afb-8bba-d2308ae8dca7","Type":"ContainerDied","Data":"15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96"}
Oct 31 01:27:43.351127 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.351061    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"
Oct 31 01:27:43.351233 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.351066    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5" event={"ID":"a5bab393-518a-4afb-8bba-d2308ae8dca7","Type":"ContainerDied","Data":"6cf880973e5b7ec3a8c8e5374eb707f9c347f0020a2a4b15c3bd3b0cf629f860"}
Oct 31 01:27:43.358751 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.358733    4935 scope.go:122] "RemoveContainer" containerID="067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3"
Oct 31 01:27:43.359076 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:27:43.359020    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3\": container with ID starting with 067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3 not found: ID does not exist" containerID="067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3"
Oct 31 01:27:43.359184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.359086    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3"} err="failed to get container status \"067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3\": rpc error: code = NotFound desc = could not find container \"067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3\": container with ID starting with 067e6e7218b84cb3c75f557d7a465e18e5a21b66c7058f98ba5b50a1b12d5cc3 not found: ID does not exist"
Oct 31 01:27:43.359184 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.359111    4935 scope.go:122] "RemoveContainer" containerID="15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96"
Oct 31 01:27:43.367958 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.367780    4935 scope.go:122] "RemoveContainer" containerID="15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96"
Oct 31 01:27:43.370024 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:27:43.370001    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96\": container with ID starting with 15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96 not found: ID does not exist" containerID="15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96"
Oct 31 01:27:43.370142 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.370029    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96"} err="failed to get container status \"15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96\": rpc error: code = NotFound desc = could not find container \"15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96\": container with ID starting with 15e0d04729c89be9f224205c71915c5c94db7e5c01320750b31a69f0d7025a96 not found: ID does not exist"
Oct 31 01:27:43.508461 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.508430    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"]
Oct 31 01:27:43.554976 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.554950    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-qrgs4"]
Oct 31 01:27:43.625462 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.625403    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"]
Oct 31 01:27:43.631321 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.631300    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-mkpp5"]
Oct 31 01:27:43.944733 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.944664    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a308a56d-7aef-45de-b856-0f8ac8c475fd" path="/var/lib/kubelet/pods/a308a56d-7aef-45de-b856-0f8ac8c475fd/volumes"
Oct 31 01:27:43.945092 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:43.945077    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bab393-518a-4afb-8bba-d2308ae8dca7" path="/var/lib/kubelet/pods/a5bab393-518a-4afb-8bba-d2308ae8dca7/volumes"
Oct 31 01:27:44.151228 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:27:44.151197    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:27:48.562276 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:48.562237    4935 scope.go:122] "RemoveContainer" containerID="4de71af938be03a8a9314c8a2f2db80ce7417fec5edcfddb1b130782a0069c97"
Oct 31 01:27:50.665728 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:27:50.665676    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 31 01:28:00.665795 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:00.665705    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 31 01:28:10.675363 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:10.675333    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:28:10.682559 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:10.682536    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:28:14.151031 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:28:14.150996    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:28:20.642540 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:20.642508    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 31 01:28:35.665525 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:35.665484    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main" containerID="cri-o://b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef" gracePeriod=45
Oct 31 01:28:36.330718 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.330695    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:28:36.342093 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.342074    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-home\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-home\") pod \"6432403d-d204-441e-a1c1-0a27fad6a1c2\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") "
Oct 31 01:28:36.342192 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.342103    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-kserve-provision-location\") pod \"6432403d-d204-441e-a1c1-0a27fad6a1c2\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") "
Oct 31 01:28:36.342192 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.342162    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-tmp-dir\") pod \"6432403d-d204-441e-a1c1-0a27fad6a1c2\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") "
Oct 31 01:28:36.342192 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.342183    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6432403d-d204-441e-a1c1-0a27fad6a1c2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6432403d-d204-441e-a1c1-0a27fad6a1c2-cabundle-cert\") pod \"6432403d-d204-441e-a1c1-0a27fad6a1c2\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") "
Oct 31 01:28:36.342330 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.342211    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-dshm\") pod \"6432403d-d204-441e-a1c1-0a27fad6a1c2\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") "
Oct 31 01:28:36.342330 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.342246    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-model-cache\") pod \"6432403d-d204-441e-a1c1-0a27fad6a1c2\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") "
Oct 31 01:28:36.342330 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.342285    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6432403d-d204-441e-a1c1-0a27fad6a1c2-kube-api-access-rkw8t\" (UniqueName: \"kubernetes.io/projected/6432403d-d204-441e-a1c1-0a27fad6a1c2-kube-api-access-rkw8t\") pod \"6432403d-d204-441e-a1c1-0a27fad6a1c2\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") "
Oct 31 01:28:36.342481 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.342351    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6432403d-d204-441e-a1c1-0a27fad6a1c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/6432403d-d204-441e-a1c1-0a27fad6a1c2-tls-certs\") pod \"6432403d-d204-441e-a1c1-0a27fad6a1c2\" (UID: \"6432403d-d204-441e-a1c1-0a27fad6a1c2\") "
Oct 31 01:28:36.342815 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.342605    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6432403d-d204-441e-a1c1-0a27fad6a1c2-cabundle-cert" pod "6432403d-d204-441e-a1c1-0a27fad6a1c2" (UID: "6432403d-d204-441e-a1c1-0a27fad6a1c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:28:36.342815 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.342784    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-model-cache" pod "6432403d-d204-441e-a1c1-0a27fad6a1c2" (UID: "6432403d-d204-441e-a1c1-0a27fad6a1c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:28:36.342975 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.342833    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-home" pod "6432403d-d204-441e-a1c1-0a27fad6a1c2" (UID: "6432403d-d204-441e-a1c1-0a27fad6a1c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:28:36.344971 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.344949    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6432403d-d204-441e-a1c1-0a27fad6a1c2-kube-api-access-rkw8t" pod "6432403d-d204-441e-a1c1-0a27fad6a1c2" (UID: "6432403d-d204-441e-a1c1-0a27fad6a1c2"). InnerVolumeSpecName "kube-api-access-rkw8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:28:36.345138 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.344992    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-dshm" pod "6432403d-d204-441e-a1c1-0a27fad6a1c2" (UID: "6432403d-d204-441e-a1c1-0a27fad6a1c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:28:36.345138 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.345059    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6432403d-d204-441e-a1c1-0a27fad6a1c2-tls-certs" pod "6432403d-d204-441e-a1c1-0a27fad6a1c2" (UID: "6432403d-d204-441e-a1c1-0a27fad6a1c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:28:36.355218 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.355192    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-tmp-dir" pod "6432403d-d204-441e-a1c1-0a27fad6a1c2" (UID: "6432403d-d204-441e-a1c1-0a27fad6a1c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:28:36.369111 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.369078    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-kserve-provision-location" pod "6432403d-d204-441e-a1c1-0a27fad6a1c2" (UID: "6432403d-d204-441e-a1c1-0a27fad6a1c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:28:36.443172 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.443124    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:36.443172 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.443145    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6432403d-d204-441e-a1c1-0a27fad6a1c2-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:36.443172 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.443155    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:36.443172 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.443163    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:36.443344 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.443173    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkw8t\" (UniqueName: \"kubernetes.io/projected/6432403d-d204-441e-a1c1-0a27fad6a1c2-kube-api-access-rkw8t\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:36.443344 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.443183    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6432403d-d204-441e-a1c1-0a27fad6a1c2-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:36.443344 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.443192    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:36.443344 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.443201    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6432403d-d204-441e-a1c1-0a27fad6a1c2-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:36.590376 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.590346    4935 generic.go:356] "Generic (PLEG): container finished" podID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerID="b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef" exitCode=0
Oct 31 01:28:36.590472 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.590394    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Oct 31 01:28:36.590472 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.590395    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6432403d-d204-441e-a1c1-0a27fad6a1c2","Type":"ContainerDied","Data":"b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef"}
Oct 31 01:28:36.590472 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.590447    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"6432403d-d204-441e-a1c1-0a27fad6a1c2","Type":"ContainerDied","Data":"1621817b3c92422ee0c982ab4def096b3289b37d5a49669cee1fb142194acde0"}
Oct 31 01:28:36.590472 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.590471    4935 scope.go:122] "RemoveContainer" containerID="b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef"
Oct 31 01:28:36.599491 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.599475    4935 scope.go:122] "RemoveContainer" containerID="57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9"
Oct 31 01:28:36.606859 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.606841    4935 scope.go:122] "RemoveContainer" containerID="b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef"
Oct 31 01:28:36.607129 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:28:36.607107    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef\": container with ID starting with b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef not found: ID does not exist" containerID="b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef"
Oct 31 01:28:36.607221 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.607132    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef"} err="failed to get container status \"b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef\": rpc error: code = NotFound desc = could not find container \"b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef\": container with ID starting with b9affa6b877ccfa242c2fbe6cd2df17e5ab4a63f400ec20719bafae24db4fcef not found: ID does not exist"
Oct 31 01:28:36.607221 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.607148    4935 scope.go:122] "RemoveContainer" containerID="57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9"
Oct 31 01:28:36.607388 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:28:36.607372    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9\": container with ID starting with 57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9 not found: ID does not exist" containerID="57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9"
Oct 31 01:28:36.607431 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.607392    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9"} err="failed to get container status \"57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9\": rpc error: code = NotFound desc = could not find container \"57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9\": container with ID starting with 57f0fb80b60eb08ccd89590a9aef1d4d598dd911b56697079a239e57be209fd9 not found: ID does not exist"
Oct 31 01:28:36.749695 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.749669    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 31 01:28:36.757212 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:36.757183    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Oct 31 01:28:37.942533 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:37.942502    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" path="/var/lib/kubelet/pods/6432403d-d204-441e-a1c1-0a27fad6a1c2/volumes"
Oct 31 01:28:44.151877 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:28:44.151834    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:28:45.600606 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.600570    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"]
Oct 31 01:28:45.600968 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.600934    4935 cpu_manager.go:414] "containerMap: removing container" podUID="a308a56d-7aef-45de-b856-0f8ac8c475fd" containerName="istio-proxy"
Oct 31 01:28:45.600968 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.600947    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a308a56d-7aef-45de-b856-0f8ac8c475fd" containerName="istio-proxy"
Oct 31 01:28:45.600968 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.600961    4935 cpu_manager.go:414] "containerMap: removing container" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="storage-initializer"
Oct 31 01:28:45.600968 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.600966    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="storage-initializer"
Oct 31 01:28:45.601125 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.600974    4935 cpu_manager.go:414] "containerMap: removing container" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main"
Oct 31 01:28:45.601125 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.600978    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main"
Oct 31 01:28:45.601125 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.600990    4935 cpu_manager.go:414] "containerMap: removing container" podUID="a5bab393-518a-4afb-8bba-d2308ae8dca7" containerName="istio-proxy"
Oct 31 01:28:45.601125 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.600995    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bab393-518a-4afb-8bba-d2308ae8dca7" containerName="istio-proxy"
Oct 31 01:28:45.601125 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.601079    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="a308a56d-7aef-45de-b856-0f8ac8c475fd" containerName="istio-proxy"
Oct 31 01:28:45.601125 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.601092    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="6432403d-d204-441e-a1c1-0a27fad6a1c2" containerName="main"
Oct 31 01:28:45.601125 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.601099    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bab393-518a-4afb-8bba-d2308ae8dca7" containerName="istio-proxy"
Oct 31 01:28:45.603795 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.603772    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.653758 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.653722    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:28:45.653924 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.653781    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"openshift-service-ca.crt\""
Oct 31 01:28:45.669245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.669225    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"kube-root-ca.crt\""
Oct 31 01:28:45.691905 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.691881    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"model-routing-only-gateway-openshift-default-dockercfg-tb42c\""
Oct 31 01:28:45.701959 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.701932    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"]
Oct 31 01:28:45.705538 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.705518    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.705630 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.705561    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-token\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.705630 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.705579    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.705630 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.705601    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-data\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.705802 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.705648    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttptk\" (UniqueName: \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-kube-api-access-ttptk\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.705802 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.705785    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82142e5f-4d6e-4feb-be27-157d733bef6f-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.705891 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.705831    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.705925 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.705900    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.705955 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.705939    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.806800 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.806774    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-token\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.806923 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.806802    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.806923 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.806823    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-data\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.806923 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.806838    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttptk\" (UniqueName: \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-kube-api-access-ttptk\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.806923 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.806871    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82142e5f-4d6e-4feb-be27-157d733bef6f-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.807173 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.807010    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.807173 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.807077    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.807173 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.807130    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.807173 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.807159    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.807364 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.807213    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.807364 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.807273    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-data\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.807723 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.807701    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82142e5f-4d6e-4feb-be27-157d733bef6f-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.807812 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.807773    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.807812 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.807786    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.809241 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.809220    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.809523 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.809507    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.866755 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.866701    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-token\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.866943 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.866924    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttptk\" (UniqueName: \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-kube-api-access-ttptk\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:45.915152 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:45.915132    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:46.338440 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:46.338408    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"]
Oct 31 01:28:46.344827 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:28:46.344800    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82142e5f_4d6e_4feb_be27_157d733bef6f.slice/crio-4cf5cd688dbfa1356d98fb03efd8d9bd8790aa0df6c2f953477e96ff6f1fb407 WatchSource:0}: Error finding container 4cf5cd688dbfa1356d98fb03efd8d9bd8790aa0df6c2f953477e96ff6f1fb407: Status 404 returned error can't find the container with id 4cf5cd688dbfa1356d98fb03efd8d9bd8790aa0df6c2f953477e96ff6f1fb407
Oct 31 01:28:46.346750 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:46.346702    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:28:46.346856 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:46.346781    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:28:46.346856 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:46.346811    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:28:46.637195 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:46.637109    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" event={"ID":"82142e5f-4d6e-4feb-be27-157d733bef6f","Type":"ContainerStarted","Data":"14aeb67a2cb514c72da62869f84af5564c51fda1900111d979c4e2dc39c36526"}
Oct 31 01:28:46.637195 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:46.637170    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" event={"ID":"82142e5f-4d6e-4feb-be27-157d733bef6f","Type":"ContainerStarted","Data":"4cf5cd688dbfa1356d98fb03efd8d9bd8790aa0df6c2f953477e96ff6f1fb407"}
Oct 31 01:28:46.754195 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:46.754143    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" podStartSLOduration=1.754129502 podStartE2EDuration="1.754129502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:46.752441567 +0000 UTC m=+3598.934970863" watchObservedRunningTime="2026-10-31 01:28:46.754129502 +0000 UTC m=+3598.936658796"
Oct 31 01:28:46.915816 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:46.915747    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:47.919836 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:47.919810    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:48.299025 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:48.299000    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:28:48.314279 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:48.314258    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:28:48.645601 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:48.645576    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:48.646497 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:48.646480    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:28:53.393489 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.393458    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"]
Oct 31 01:28:53.396749 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.396731    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.467345 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.467310    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-tmp-dir\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.467479 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.467354    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-dshm\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.467479 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.467432    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d508eea-211f-443f-94f0-1ec32ce963de-tls-certs\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.467550 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.467488    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-home\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.467649 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.467625    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-model-cache\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.467698 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.467680    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbjr\" (UniqueName: \"kubernetes.io/projected/6d508eea-211f-443f-94f0-1ec32ce963de-kube-api-access-dlbjr\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.492923 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.492895    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"default-dockercfg-c2f8m\""
Oct 31 01:28:53.492923 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.492912    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"model-routing-only-7aad4d71-kserve-self-signed-certs\""
Oct 31 01:28:53.494756 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.494733    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"]
Oct 31 01:28:53.568190 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.568163    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-model-cache\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.568307 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.568206    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbjr\" (UniqueName: \"kubernetes.io/projected/6d508eea-211f-443f-94f0-1ec32ce963de-kube-api-access-dlbjr\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.568307 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.568245    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-tmp-dir\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.568307 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.568283    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-dshm\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.568470 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.568307    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d508eea-211f-443f-94f0-1ec32ce963de-tls-certs\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.568470 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.568348    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-home\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.568583 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.568564    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-model-cache\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.568645 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.568600    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-tmp-dir\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.568750 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.568731    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-home\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.570585 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.570554    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-dshm\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.570692 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.570673    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d508eea-211f-443f-94f0-1ec32ce963de-tls-certs\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.627331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.627307    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbjr\" (UniqueName: \"kubernetes.io/projected/6d508eea-211f-443f-94f0-1ec32ce963de-kube-api-access-dlbjr\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:53.708288 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:53.708238    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:54.081233 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:54.081201    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"]
Oct 31 01:28:54.087487 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:28:54.087460    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d508eea_211f_443f_94f0_1ec32ce963de.slice/crio-f2382d52137791c479dd0262d272a2cc2f7639f0143f43d7327d354b606e6036 WatchSource:0}: Error finding container f2382d52137791c479dd0262d272a2cc2f7639f0143f43d7327d354b606e6036: Status 404 returned error can't find the container with id f2382d52137791c479dd0262d272a2cc2f7639f0143f43d7327d354b606e6036
Oct 31 01:28:54.672721 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:54.672651    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9" event={"ID":"6d508eea-211f-443f-94f0-1ec32ce963de","Type":"ContainerStarted","Data":"38c7b4389fcdf5072a04880598ac2773fcb3ba5fc038bea29140793a2d1933a3"}
Oct 31 01:28:54.673094 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:54.672745    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9" event={"ID":"6d508eea-211f-443f-94f0-1ec32ce963de","Type":"ContainerStarted","Data":"f2382d52137791c479dd0262d272a2cc2f7639f0143f43d7327d354b606e6036"}
Oct 31 01:28:54.708695 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:54.708674    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:28:54.786366 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:28:54.786318    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9" podStartSLOduration=1.786305024 podStartE2EDuration="1.786305024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:54.784583055 +0000 UTC m=+3606.967112349" watchObservedRunningTime="2026-10-31 01:28:54.786305024 +0000 UTC m=+3606.968834317"
Oct 31 01:29:03.708482 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:03.708438    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:29:03.721405 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:03.721372    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:29:04.687194 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:04.687150    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:29:04.720648 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:04.720618    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:29:06.726787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:06.726759    4935 generic.go:356] "Generic (PLEG): container finished" podID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerID="14aeb67a2cb514c72da62869f84af5564c51fda1900111d979c4e2dc39c36526" exitCode=0
Oct 31 01:29:06.727350 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:06.726797    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" event={"ID":"82142e5f-4d6e-4feb-be27-157d733bef6f","Type":"ContainerDied","Data":"14aeb67a2cb514c72da62869f84af5564c51fda1900111d979c4e2dc39c36526"}
Oct 31 01:29:06.727350 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:06.727228    4935 scope.go:122] "RemoveContainer" containerID="14aeb67a2cb514c72da62869f84af5564c51fda1900111d979c4e2dc39c36526"
Oct 31 01:29:06.728225 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:06.728196    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:29:06.728310 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:06.728263    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:29:06.728310 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:06.728299    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:29:07.733511 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:07.733472    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" event={"ID":"82142e5f-4d6e-4feb-be27-157d733bef6f","Type":"ContainerStarted","Data":"b48a3e8299b31f1416cc68c556bd1b30ee3417226023c35525a4eeafdf9ed748"}
Oct 31 01:29:07.915720 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:07.915683    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:29:07.915997 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:07.915968    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 31 01:29:08.915755 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:08.915715    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 31 01:29:09.915901 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:09.915861    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 31 01:29:10.916216 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:10.916177    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 31 01:29:11.915617 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:11.915579    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 31 01:29:12.918712 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:12.918683    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:29:12.919114 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:12.919063    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:29:12.919666 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:12.919642    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:29:14.151607 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:29:14.151573    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:29:18.350145 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.350114    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"]
Oct 31 01:29:18.355616 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.355590    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.394792 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.394766    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"]
Oct 31 01:29:18.407103 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.407071    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Oct 31 01:29:18.407266 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.407119    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-b668w\""
Oct 31 01:29:18.439451 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.439423    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Oct 31 01:29:18.439582 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.439491    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Oct 31 01:29:18.439582 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.439493    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:29:18.439582 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.439502    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Oct 31 01:29:18.473850 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.473827    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-tmp-dir\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.473952 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.473860    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-model-cache\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.473952 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.473878    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f576b0a5-98fc-4175-9533-50734f403e0f-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.474066 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.473943    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.474066 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.473975    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-dshm\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.474066 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.473998    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f576b0a5-98fc-4175-9533-50734f403e0f-tls-certs\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.474176 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.474078    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq8bh\" (UniqueName: \"kubernetes.io/projected/f576b0a5-98fc-4175-9533-50734f403e0f-kube-api-access-sq8bh\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.474176 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.474107    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-home\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.575392 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575366    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.575516 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575398    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-dshm\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.575516 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575419    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f576b0a5-98fc-4175-9533-50734f403e0f-tls-certs\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.575516 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575437    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq8bh\" (UniqueName: \"kubernetes.io/projected/f576b0a5-98fc-4175-9533-50734f403e0f-kube-api-access-sq8bh\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.575516 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575461    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-home\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.575516 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575509    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-tmp-dir\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.575771 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575532    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-model-cache\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.575771 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575548    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f576b0a5-98fc-4175-9533-50734f403e0f-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.575885 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575823    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.575885 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575855    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-tmp-dir\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.575987 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575923    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-home\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.576023 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.575986    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-model-cache\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.576401 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.576335    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f576b0a5-98fc-4175-9533-50734f403e0f-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.577925 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.577904    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-dshm\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.578068 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.578050    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f576b0a5-98fc-4175-9533-50734f403e0f-tls-certs\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.634931 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.634882    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq8bh\" (UniqueName: \"kubernetes.io/projected/f576b0a5-98fc-4175-9533-50734f403e0f-kube-api-access-sq8bh\") pod \"tls-verification-test-kserve-prefill-55567c6db7-grzct\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:18.673845 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:18.673823    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:19.101166 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:19.101137    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"]
Oct 31 01:29:19.106250 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:29:19.106224    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf576b0a5_98fc_4175_9533_50734f403e0f.slice/crio-8690d58782217d1df6d37a60f08e0c614596eb3a02caeb97863c92c262d0a11d WatchSource:0}: Error finding container 8690d58782217d1df6d37a60f08e0c614596eb3a02caeb97863c92c262d0a11d: Status 404 returned error can't find the container with id 8690d58782217d1df6d37a60f08e0c614596eb3a02caeb97863c92c262d0a11d
Oct 31 01:29:19.788273 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:19.788237    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" event={"ID":"f576b0a5-98fc-4175-9533-50734f403e0f","Type":"ContainerStarted","Data":"9954eb1910806f0e8d860770dcd55691f2444f632e5b6421e32fb278d1fe8ca0"}
Oct 31 01:29:19.788674 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:19.788292    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" event={"ID":"f576b0a5-98fc-4175-9533-50734f403e0f","Type":"ContainerStarted","Data":"8690d58782217d1df6d37a60f08e0c614596eb3a02caeb97863c92c262d0a11d"}
Oct 31 01:29:21.800759 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:21.800725    4935 generic.go:356] "Generic (PLEG): container finished" podID="f576b0a5-98fc-4175-9533-50734f403e0f" containerID="9954eb1910806f0e8d860770dcd55691f2444f632e5b6421e32fb278d1fe8ca0" exitCode=0
Oct 31 01:29:21.801180 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:21.800799    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" event={"ID":"f576b0a5-98fc-4175-9533-50734f403e0f","Type":"ContainerDied","Data":"9954eb1910806f0e8d860770dcd55691f2444f632e5b6421e32fb278d1fe8ca0"}
Oct 31 01:29:22.808310 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:22.808264    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" event={"ID":"f576b0a5-98fc-4175-9533-50734f403e0f","Type":"ContainerStarted","Data":"19b98bfd674abcc78458e790829da247268edb24735753a7e0202c3d27d6f1a4"}
Oct 31 01:29:23.022591 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:23.022541    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podStartSLOduration=2.968242163 podStartE2EDuration="5.022529706s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.054287543s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:22.975539382 +0000 UTC m=+3635.158068691" watchObservedRunningTime="2026-10-31 01:29:23.022529706 +0000 UTC m=+3635.205059051"
Oct 31 01:29:28.674394 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:28.674316    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:28.674394 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:28.674386    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:29:28.675922 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:28.675892    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Oct 31 01:29:37.692030 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.691988    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"]
Oct 31 01:29:37.696819 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.696792    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.739297 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.739255    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"]
Oct 31 01:29:37.743484 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.743465    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.791545 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.791516    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"]
Oct 31 01:29:37.849243 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.849191    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"]
Oct 31 01:29:37.866255 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866229    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.866406 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866262    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.866406 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866286    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.866406 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866371    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.866406 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866396    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e950da-255b-405e-9bf0-621c4f39c959-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.866585 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866421    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.866585 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866437    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e950da-255b-405e-9bf0-621c4f39c959-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.866654 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866608    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.866699 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866663    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.866736 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866726    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.866821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866800    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.866857 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866842    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skbwp\" (UniqueName: \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-kube-api-access-skbwp\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.866935 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866920    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.866972 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866953    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e950da-255b-405e-9bf0-621c4f39c959-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.867007 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.866991    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.867080 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.867017    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.867080 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.867060    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zghxv\" (UniqueName: \"kubernetes.io/projected/42e950da-255b-405e-9bf0-621c4f39c959-kube-api-access-zghxv\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.867193 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.867136    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.968240 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968156    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.968240 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968190    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e950da-255b-405e-9bf0-621c4f39c959-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.968240 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968218    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.968524 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968243    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.968524 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968266    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zghxv\" (UniqueName: \"kubernetes.io/projected/42e950da-255b-405e-9bf0-621c4f39c959-kube-api-access-zghxv\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.968524 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968292    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.968524 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968360    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.968524 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968396    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.968524 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968431    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.968524 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968476    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.968524 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968501    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e950da-255b-405e-9bf0-621c4f39c959-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.968969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968542    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.968969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968569    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e950da-255b-405e-9bf0-621c4f39c959-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.968969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968617    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.968969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968617    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.968969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968663    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.968969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968675    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.968969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968679    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.968969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968745    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.968969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968780    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.968969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968813    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skbwp\" (UniqueName: \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-kube-api-access-skbwp\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.969539 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.968990    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.969539 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.969121    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.969539 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.969295    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.969539 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.969309    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.969539 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.969377    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.969820 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.969571    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.969927 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.969907    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e950da-255b-405e-9bf0-621c4f39c959-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.971666 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.971643    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e950da-255b-405e-9bf0-621c4f39c959-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.971949 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.971920    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:37.972201 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.972175    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e950da-255b-405e-9bf0-621c4f39c959-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:37.972446 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:37.972387    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:38.027713 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.027689    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e950da-255b-405e-9bf0-621c4f39c959-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:38.028228 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.028194    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skbwp\" (UniqueName: \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-kube-api-access-skbwp\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:38.028406 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.028384    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-kb28w\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:38.028510 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.028493    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zghxv\" (UniqueName: \"kubernetes.io/projected/42e950da-255b-405e-9bf0-621c4f39c959-kube-api-access-zghxv\") pod \"openshift-ai-inference-openshift-default-568c495856-q9tdb\" (UID: \"42e950da-255b-405e-9bf0-621c4f39c959\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:38.062329 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.062303    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:38.310253 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.310222    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:38.492363 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.492320    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"]
Oct 31 01:29:38.498204 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:29:38.498179    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e950da_255b_405e_9bf0_621c4f39c959.slice/crio-cdf7ad035b5008d41684f7ef39a975121154ecf01a89a46dc4df5eee8583411a WatchSource:0}: Error finding container cdf7ad035b5008d41684f7ef39a975121154ecf01a89a46dc4df5eee8583411a: Status 404 returned error can't find the container with id cdf7ad035b5008d41684f7ef39a975121154ecf01a89a46dc4df5eee8583411a
Oct 31 01:29:38.500395 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.500354    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:29:38.500473 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.500445    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:29:38.500508 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.500488    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:29:38.674815 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.674725    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Oct 31 01:29:38.684195 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.684165    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"]
Oct 31 01:29:38.690921 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:29:38.690888    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf49fc49_8332_42c1_9ce5_e62fd20fc666.slice/crio-44fc6243c9725a874f281719e3fffe94ae50e42509269ae97824137b101fd3b9 WatchSource:0}: Error finding container 44fc6243c9725a874f281719e3fffe94ae50e42509269ae97824137b101fd3b9: Status 404 returned error can't find the container with id 44fc6243c9725a874f281719e3fffe94ae50e42509269ae97824137b101fd3b9
Oct 31 01:29:38.693093 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.693063    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:29:38.693500 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.693138    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:29:38.693500 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.693183    4935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 31 01:29:38.896245 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.896206    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" event={"ID":"bf49fc49-8332-42c1-9ce5-e62fd20fc666","Type":"ContainerStarted","Data":"b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc"}
Oct 31 01:29:38.900153 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.900109    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" event={"ID":"bf49fc49-8332-42c1-9ce5-e62fd20fc666","Type":"ContainerStarted","Data":"44fc6243c9725a874f281719e3fffe94ae50e42509269ae97824137b101fd3b9"}
Oct 31 01:29:38.900401 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.900381    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb" event={"ID":"42e950da-255b-405e-9bf0-621c4f39c959","Type":"ContainerStarted","Data":"79d411dd2302465f22713e9e69e103ecb46e2b056ee8522e62adcdcb319800a9"}
Oct 31 01:29:38.900527 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:38.900511    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb" event={"ID":"42e950da-255b-405e-9bf0-621c4f39c959","Type":"ContainerStarted","Data":"cdf7ad035b5008d41684f7ef39a975121154ecf01a89a46dc4df5eee8583411a"}
Oct 31 01:29:39.018316 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:39.018262    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" podStartSLOduration=2.018246669 podStartE2EDuration="2.018246669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:39.017061097 +0000 UTC m=+3651.199590388" watchObservedRunningTime="2026-10-31 01:29:39.018246669 +0000 UTC m=+3651.200775988"
Oct 31 01:29:39.067740 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:39.067708    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:39.071057 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:39.070137    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-q9tdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:29:39.071057 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:39.070197    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb" podUID="42e950da-255b-405e-9bf0-621c4f39c959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 01:29:39.141972 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:39.141895    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb" podStartSLOduration=2.141875513 podStartE2EDuration="2.141875513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:39.140654231 +0000 UTC m=+3651.323183525" watchObservedRunningTime="2026-10-31 01:29:39.141875513 +0000 UTC m=+3651.324404809"
Oct 31 01:29:39.311299 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:39.311192    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:39.312861 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:39.312815    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kb28w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:29:39.313008 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:39.312893    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" podUID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:29:40.067062 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:40.063494    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-q9tdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:29:40.067062 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:40.063557    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb" podUID="42e950da-255b-405e-9bf0-621c4f39c959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 01:29:40.311516 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:40.311477    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kb28w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:29:40.311692 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:40.311531    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" podUID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:29:41.064331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:41.064010    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-q9tdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:29:41.064331 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:41.064103    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb" podUID="42e950da-255b-405e-9bf0-621c4f39c959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 01:29:41.312059 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:41.312005    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kb28w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:29:41.312557 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:41.312099    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" podUID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:29:42.064415 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:42.064374    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-q9tdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:29:42.064724 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:42.064686    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb" podUID="42e950da-255b-405e-9bf0-621c4f39c959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 01:29:42.311308 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:42.311183    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kb28w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:29:42.311308 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:42.311228    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" podUID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:29:43.063450 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:43.063410    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-q9tdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:29:43.064095 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:43.064059    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb" podUID="42e950da-255b-405e-9bf0-621c4f39c959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 01:29:43.311643 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:43.311590    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kb28w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:29:43.311848 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:43.311655    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" podUID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:29:44.065439 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:44.065398    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-q9tdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:29:44.066002 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:44.065969    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb" podUID="42e950da-255b-405e-9bf0-621c4f39c959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 31 01:29:44.175447 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:29:44.175410    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:29:44.311027 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:44.310994    4935 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-kb28w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:29:44.311247 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:44.311061    4935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" podUID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:29:45.066866 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:45.066833    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:45.067345 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:45.067325    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:45.067978 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:45.067964    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-q9tdb"
Oct 31 01:29:45.314760 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:45.314731    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:45.315067 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:45.315023    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:45.315831 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:45.315810    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:29:48.674360 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:48.674315    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Oct 31 01:29:58.674496 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:29:58.674453    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Oct 31 01:30:08.675000 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:30:08.674960    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Oct 31 01:30:14.151771 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:30:14.151730    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:30:18.675269 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:30:18.675230    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Oct 31 01:30:28.674222 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:30:28.674175    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Oct 31 01:30:38.674413 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:30:38.674369    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Oct 31 01:30:44.152616 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:30:44.152578    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:30:48.608115 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:30:48.608078    4935 scope.go:122] "RemoveContainer" containerID="90b9c60fcd313f7652ef161b49f88d71b4fdcea4740ce0a4a162c56599836118"
Oct 31 01:30:48.674851 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:30:48.674817    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Oct 31 01:30:58.674830 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:30:58.674739    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Oct 31 01:31:08.684435 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:08.684402    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:31:08.691792 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:08.691767    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:31:14.152018 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:31:14.151978    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:31:17.100969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.100933    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"]
Oct 31 01:31:17.110417 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:31:17.110381    4935 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="38c7b4389fcdf5072a04880598ac2773fcb3ba5fc038bea29140793a2d1933a3" cmd=["/bin/sleep","15"]
Oct 31 01:31:17.110549 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.110433    4935 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 01:31:17.110549 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:31:17.110444    4935 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9" podUID="6d508eea-211f-443f-94f0-1ec32ce963de" containerName="main" containerID="cri-o://38c7b4389fcdf5072a04880598ac2773fcb3ba5fc038bea29140793a2d1933a3"
Oct 31 01:31:17.110549 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.110493    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9" podUID="6d508eea-211f-443f-94f0-1ec32ce963de" containerName="main" containerID="cri-o://38c7b4389fcdf5072a04880598ac2773fcb3ba5fc038bea29140793a2d1933a3" gracePeriod=60
Oct 31 01:31:17.150640 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.150611    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"]
Oct 31 01:31:17.150955 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.150925    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy" containerID="cri-o://b48a3e8299b31f1416cc68c556bd1b30ee3417226023c35525a4eeafdf9ed748" gracePeriod=30
Oct 31 01:31:17.387901 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.387838    4935 generic.go:356] "Generic (PLEG): container finished" podID="6d508eea-211f-443f-94f0-1ec32ce963de" containerID="38c7b4389fcdf5072a04880598ac2773fcb3ba5fc038bea29140793a2d1933a3" exitCode=0
Oct 31 01:31:17.387901 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.387887    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9" event={"ID":"6d508eea-211f-443f-94f0-1ec32ce963de","Type":"ContainerDied","Data":"38c7b4389fcdf5072a04880598ac2773fcb3ba5fc038bea29140793a2d1933a3"}
Oct 31 01:31:17.721368 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.721298    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9" podUID="6d508eea-211f-443f-94f0-1ec32ce963de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Oct 31 01:31:17.854426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.854403    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:31:17.874464 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.874442    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d508eea-211f-443f-94f0-1ec32ce963de-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d508eea-211f-443f-94f0-1ec32ce963de-tls-certs\") pod \"6d508eea-211f-443f-94f0-1ec32ce963de\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") "
Oct 31 01:31:17.874566 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.874484    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-home\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-home\") pod \"6d508eea-211f-443f-94f0-1ec32ce963de\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") "
Oct 31 01:31:17.874566 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.874520    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d508eea-211f-443f-94f0-1ec32ce963de-kube-api-access-dlbjr\" (UniqueName: \"kubernetes.io/projected/6d508eea-211f-443f-94f0-1ec32ce963de-kube-api-access-dlbjr\") pod \"6d508eea-211f-443f-94f0-1ec32ce963de\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") "
Oct 31 01:31:17.874566 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.874561    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-dshm\") pod \"6d508eea-211f-443f-94f0-1ec32ce963de\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") "
Oct 31 01:31:17.874736 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.874578    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-model-cache\") pod \"6d508eea-211f-443f-94f0-1ec32ce963de\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") "
Oct 31 01:31:17.874736 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.874609    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-tmp-dir\") pod \"6d508eea-211f-443f-94f0-1ec32ce963de\" (UID: \"6d508eea-211f-443f-94f0-1ec32ce963de\") "
Oct 31 01:31:17.874849 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.874819    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-home" pod "6d508eea-211f-443f-94f0-1ec32ce963de" (UID: "6d508eea-211f-443f-94f0-1ec32ce963de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:17.874933 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.874885    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-model-cache" pod "6d508eea-211f-443f-94f0-1ec32ce963de" (UID: "6d508eea-211f-443f-94f0-1ec32ce963de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:17.875060 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.874951    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-tmp-dir" pod "6d508eea-211f-443f-94f0-1ec32ce963de" (UID: "6d508eea-211f-443f-94f0-1ec32ce963de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:17.876702 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.876683    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d508eea-211f-443f-94f0-1ec32ce963de-tls-certs" pod "6d508eea-211f-443f-94f0-1ec32ce963de" (UID: "6d508eea-211f-443f-94f0-1ec32ce963de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:31:17.876844 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.876827    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-dshm" pod "6d508eea-211f-443f-94f0-1ec32ce963de" (UID: "6d508eea-211f-443f-94f0-1ec32ce963de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:17.876936 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.876917    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d508eea-211f-443f-94f0-1ec32ce963de-kube-api-access-dlbjr" pod "6d508eea-211f-443f-94f0-1ec32ce963de" (UID: "6d508eea-211f-443f-94f0-1ec32ce963de"). InnerVolumeSpecName "kube-api-access-dlbjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:17.963198 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:31:17.963176    4935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d508eea_211f_443f_94f0_1ec32ce963de.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 01:31:17.976131 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.976068    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:17.976131 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.976116    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d508eea-211f-443f-94f0-1ec32ce963de-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:17.976131 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.976131    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:17.976715 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.976146    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbjr\" (UniqueName: \"kubernetes.io/projected/6d508eea-211f-443f-94f0-1ec32ce963de-kube-api-access-dlbjr\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:17.976715 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.976190    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:17.976715 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:17.976421    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d508eea-211f-443f-94f0-1ec32ce963de-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:18.393433 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:18.393398    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"
Oct 31 01:31:18.393868 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:18.393439    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9" event={"ID":"6d508eea-211f-443f-94f0-1ec32ce963de","Type":"ContainerDied","Data":"f2382d52137791c479dd0262d272a2cc2f7639f0143f43d7327d354b606e6036"}
Oct 31 01:31:18.393868 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:18.393507    4935 scope.go:122] "RemoveContainer" containerID="38c7b4389fcdf5072a04880598ac2773fcb3ba5fc038bea29140793a2d1933a3"
Oct 31 01:31:18.550952 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:18.550926    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"]
Oct 31 01:31:18.582815 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:18.582793    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-5rdl9"]
Oct 31 01:31:19.943826 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:19.943792    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d508eea-211f-443f-94f0-1ec32ce963de" path="/var/lib/kubelet/pods/6d508eea-211f-443f-94f0-1ec32ce963de/volumes"
Oct 31 01:31:22.176820 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.176792    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"]
Oct 31 01:31:22.177248 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.177204    4935 cpu_manager.go:414] "containerMap: removing container" podUID="6d508eea-211f-443f-94f0-1ec32ce963de" containerName="main"
Oct 31 01:31:22.177248 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.177216    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d508eea-211f-443f-94f0-1ec32ce963de" containerName="main"
Oct 31 01:31:22.177388 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.177284    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d508eea-211f-443f-94f0-1ec32ce963de" containerName="main"
Oct 31 01:31:22.180389 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.180371    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.207159 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.207134    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-model-cache\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.207273 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.207170    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-home\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.207273 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.207188    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kserve-provision-location\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.207411 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.207285    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9nj2\" (UniqueName: \"kubernetes.io/projected/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kube-api-access-k9nj2\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.207411 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.207327    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tmp-dir\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.207411 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.207347    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tls-certs\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.207576 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.207430    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-cabundle-cert\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.207576 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.207446    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-dshm\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.276691 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.276663    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Oct 31 01:31:22.276841 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.276756    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Oct 31 01:31:22.276966 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.276776    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Oct 31 01:31:22.277098 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.276840    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:31:22.277098 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.276905    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4r5wb\""
Oct 31 01:31:22.277098 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.276958    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Oct 31 01:31:22.308028 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.307993    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-model-cache\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.308178 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.308084    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-home\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.308178 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.308112    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kserve-provision-location\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.308178 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.308149    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9nj2\" (UniqueName: \"kubernetes.io/projected/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kube-api-access-k9nj2\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.308294 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.308198    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tmp-dir\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.308294 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.308232    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tls-certs\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.308294 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.308287    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-cabundle-cert\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.308381 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.308308    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-dshm\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.309108 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.309081    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-model-cache\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.309301 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.309247    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tmp-dir\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.309452 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.309424    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kserve-provision-location\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.309550 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.309530    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-cabundle-cert\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.309774 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.309753    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-home\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.312505 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.312473    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tls-certs\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.312811 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.312792    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-dshm\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.373861 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.373825    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9nj2\" (UniqueName: \"kubernetes.io/projected/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kube-api-access-k9nj2\") pod \"prestop-hook-test-kserve-6578f4856b-z6qqd\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.423027 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.422947    4935 generic.go:356] "Generic (PLEG): container finished" podID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerID="b48a3e8299b31f1416cc68c556bd1b30ee3417226023c35525a4eeafdf9ed748" exitCode=0
Oct 31 01:31:22.423188 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.423025    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" event={"ID":"82142e5f-4d6e-4feb-be27-157d733bef6f","Type":"ContainerDied","Data":"b48a3e8299b31f1416cc68c556bd1b30ee3417226023c35525a4eeafdf9ed748"}
Oct 31 01:31:22.423188 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.423153    4935 scope.go:122] "RemoveContainer" containerID="14aeb67a2cb514c72da62869f84af5564c51fda1900111d979c4e2dc39c36526"
Oct 31 01:31:22.426564 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.426543    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"]
Oct 31 01:31:22.454300 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.454275    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:31:22.510644 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.510618    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82142e5f-4d6e-4feb-be27-157d733bef6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82142e5f-4d6e-4feb-be27-157d733bef6f-istiod-ca-cert\") pod \"82142e5f-4d6e-4feb-be27-157d733bef6f\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") "
Oct 31 01:31:22.510821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.510709    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-certs\") pod \"82142e5f-4d6e-4feb-be27-157d733bef6f\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") "
Oct 31 01:31:22.510821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.510768    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-socket\") pod \"82142e5f-4d6e-4feb-be27-157d733bef6f\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") "
Oct 31 01:31:22.510821 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.510797    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-token\" (UniqueName: \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-token\") pod \"82142e5f-4d6e-4feb-be27-157d733bef6f\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") "
Oct 31 01:31:22.510982 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.510823    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-podinfo\") pod \"82142e5f-4d6e-4feb-be27-157d733bef6f\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") "
Oct 31 01:31:22.510982 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.510850    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-data\") pod \"82142e5f-4d6e-4feb-be27-157d733bef6f\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") "
Oct 31 01:31:22.510982 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.510934    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-envoy\") pod \"82142e5f-4d6e-4feb-be27-157d733bef6f\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") "
Oct 31 01:31:22.510982 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.510966    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-kube-api-access-ttptk\" (UniqueName: \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-kube-api-access-ttptk\") pod \"82142e5f-4d6e-4feb-be27-157d733bef6f\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") "
Oct 31 01:31:22.511215 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.510999    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-credential-socket\") pod \"82142e5f-4d6e-4feb-be27-157d733bef6f\" (UID: \"82142e5f-4d6e-4feb-be27-157d733bef6f\") "
Oct 31 01:31:22.511215 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.511020    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82142e5f-4d6e-4feb-be27-157d733bef6f-istiod-ca-cert" pod "82142e5f-4d6e-4feb-be27-157d733bef6f" (UID: "82142e5f-4d6e-4feb-be27-157d733bef6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:22.511215 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.511126    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-certs" pod "82142e5f-4d6e-4feb-be27-157d733bef6f" (UID: "82142e5f-4d6e-4feb-be27-157d733bef6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:22.511215 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.511197    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-socket" pod "82142e5f-4d6e-4feb-be27-157d733bef6f" (UID: "82142e5f-4d6e-4feb-be27-157d733bef6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:22.511416 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.511356    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-credential-socket" pod "82142e5f-4d6e-4feb-be27-157d733bef6f" (UID: "82142e5f-4d6e-4feb-be27-157d733bef6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:22.511450 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.511407    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:22.511450 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.511431    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:22.511450 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.511446    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82142e5f-4d6e-4feb-be27-157d733bef6f-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:22.511615 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.511280    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-data" pod "82142e5f-4d6e-4feb-be27-157d733bef6f" (UID: "82142e5f-4d6e-4feb-be27-157d733bef6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:22.513488 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.513458    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-podinfo" pod "82142e5f-4d6e-4feb-be27-157d733bef6f" (UID: "82142e5f-4d6e-4feb-be27-157d733bef6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:31:22.513581 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.513470    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-kube-api-access-ttptk" pod "82142e5f-4d6e-4feb-be27-157d733bef6f" (UID: "82142e5f-4d6e-4feb-be27-157d733bef6f"). InnerVolumeSpecName "kube-api-access-ttptk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:22.513629 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.513601    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-envoy" pod "82142e5f-4d6e-4feb-be27-157d733bef6f" (UID: "82142e5f-4d6e-4feb-be27-157d733bef6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:22.514261 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.514241    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-token" pod "82142e5f-4d6e-4feb-be27-157d733bef6f" (UID: "82142e5f-4d6e-4feb-be27-157d733bef6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:22.562071 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.562047    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:22.612464 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.612427    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:22.612464 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.612461    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:22.612715 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.612475    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:22.612715 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.612489    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:22.612715 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.612503    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttptk\" (UniqueName: \"kubernetes.io/projected/82142e5f-4d6e-4feb-be27-157d733bef6f-kube-api-access-ttptk\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:22.612715 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:22.612520    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82142e5f-4d6e-4feb-be27-157d733bef6f-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:23.060651 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:23.060613    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"]
Oct 31 01:31:23.069497 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:31:23.069461    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35cc598_1b4e_4e75_a7c7_f0cb70b2d8b6.slice/crio-befdb06633002c26a5f5193ef6290e407ba27a29db44b334454349ef83ca2d3d WatchSource:0}: Error finding container befdb06633002c26a5f5193ef6290e407ba27a29db44b334454349ef83ca2d3d: Status 404 returned error can't find the container with id befdb06633002c26a5f5193ef6290e407ba27a29db44b334454349ef83ca2d3d
Oct 31 01:31:23.429889 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:23.429800    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"
Oct 31 01:31:23.429889 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:23.429844    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8" event={"ID":"82142e5f-4d6e-4feb-be27-157d733bef6f","Type":"ContainerDied","Data":"4cf5cd688dbfa1356d98fb03efd8d9bd8790aa0df6c2f953477e96ff6f1fb407"}
Oct 31 01:31:23.430469 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:23.429910    4935 scope.go:122] "RemoveContainer" containerID="b48a3e8299b31f1416cc68c556bd1b30ee3417226023c35525a4eeafdf9ed748"
Oct 31 01:31:23.431806 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:23.431775    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" event={"ID":"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6","Type":"ContainerStarted","Data":"00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef"}
Oct 31 01:31:23.431908 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:23.431831    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" event={"ID":"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6","Type":"ContainerStarted","Data":"befdb06633002c26a5f5193ef6290e407ba27a29db44b334454349ef83ca2d3d"}
Oct 31 01:31:23.720204 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:23.720123    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"]
Oct 31 01:31:23.722994 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:23.722967    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-v28x8"]
Oct 31 01:31:23.943939 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:23.943906    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" path="/var/lib/kubelet/pods/82142e5f-4d6e-4feb-be27-157d733bef6f/volumes"
Oct 31 01:31:25.443237 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:25.443206    4935 generic.go:356] "Generic (PLEG): container finished" podID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerID="00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef" exitCode=0
Oct 31 01:31:25.443585 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:25.443248    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" event={"ID":"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6","Type":"ContainerDied","Data":"00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef"}
Oct 31 01:31:26.449779 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:26.449742    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" event={"ID":"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6","Type":"ContainerStarted","Data":"5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462"}
Oct 31 01:31:26.562813 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:26.562780    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:26.660454 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:26.660404    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podStartSLOduration=2.499618087 podStartE2EDuration="4.660391076s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.160772989s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:26.613384507 +0000 UTC m=+3758.795913801" watchObservedRunningTime="2026-10-31 01:31:26.660391076 +0000 UTC m=+3758.842920367"
Oct 31 01:31:30.785084 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:30.785050    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"]
Oct 31 01:31:32.563273 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:32.563234    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:31:32.565060 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:32.565017    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 31 01:31:42.562703 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:42.562658    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 31 01:31:44.152606 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:31:44.152573    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:31:45.807827 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:45.807787    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main" containerID="cri-o://19b98bfd674abcc78458e790829da247268edb24735753a7e0202c3d27d6f1a4" gracePeriod=45
Oct 31 01:31:46.552857 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.552827    4935 generic.go:356] "Generic (PLEG): container finished" podID="f576b0a5-98fc-4175-9533-50734f403e0f" containerID="19b98bfd674abcc78458e790829da247268edb24735753a7e0202c3d27d6f1a4" exitCode=0
Oct 31 01:31:46.552995 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.552892    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" event={"ID":"f576b0a5-98fc-4175-9533-50734f403e0f","Type":"ContainerDied","Data":"19b98bfd674abcc78458e790829da247268edb24735753a7e0202c3d27d6f1a4"}
Oct 31 01:31:46.552995 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.552981    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct" event={"ID":"f576b0a5-98fc-4175-9533-50734f403e0f","Type":"ContainerDied","Data":"8690d58782217d1df6d37a60f08e0c614596eb3a02caeb97863c92c262d0a11d"}
Oct 31 01:31:46.553123 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.553005    4935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8690d58782217d1df6d37a60f08e0c614596eb3a02caeb97863c92c262d0a11d"
Oct 31 01:31:46.555618 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.555602    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:31:46.622933 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.622878    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f576b0a5-98fc-4175-9533-50734f403e0f-kube-api-access-sq8bh\" (UniqueName: \"kubernetes.io/projected/f576b0a5-98fc-4175-9533-50734f403e0f-kube-api-access-sq8bh\") pod \"f576b0a5-98fc-4175-9533-50734f403e0f\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") "
Oct 31 01:31:46.623058 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.622950    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-home\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-home\") pod \"f576b0a5-98fc-4175-9533-50734f403e0f\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") "
Oct 31 01:31:46.623058 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.622993    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f576b0a5-98fc-4175-9533-50734f403e0f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f576b0a5-98fc-4175-9533-50734f403e0f-cabundle-cert\") pod \"f576b0a5-98fc-4175-9533-50734f403e0f\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") "
Oct 31 01:31:46.623185 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.623060    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f576b0a5-98fc-4175-9533-50734f403e0f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f576b0a5-98fc-4175-9533-50734f403e0f-tls-certs\") pod \"f576b0a5-98fc-4175-9533-50734f403e0f\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") "
Oct 31 01:31:46.623185 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.623128    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-model-cache\") pod \"f576b0a5-98fc-4175-9533-50734f403e0f\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") "
Oct 31 01:31:46.623185 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.623179    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-dshm\") pod \"f576b0a5-98fc-4175-9533-50734f403e0f\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") "
Oct 31 01:31:46.623333 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.623218    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-kserve-provision-location\") pod \"f576b0a5-98fc-4175-9533-50734f403e0f\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") "
Oct 31 01:31:46.623333 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.623258    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-tmp-dir\") pod \"f576b0a5-98fc-4175-9533-50734f403e0f\" (UID: \"f576b0a5-98fc-4175-9533-50734f403e0f\") "
Oct 31 01:31:46.623426 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.623361    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f576b0a5-98fc-4175-9533-50734f403e0f-cabundle-cert" pod "f576b0a5-98fc-4175-9533-50734f403e0f" (UID: "f576b0a5-98fc-4175-9533-50734f403e0f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:46.623534 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.623505    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-model-cache" pod "f576b0a5-98fc-4175-9533-50734f403e0f" (UID: "f576b0a5-98fc-4175-9533-50734f403e0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:46.623594 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.623519    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f576b0a5-98fc-4175-9533-50734f403e0f-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.624061 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.624013    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-home" pod "f576b0a5-98fc-4175-9533-50734f403e0f" (UID: "f576b0a5-98fc-4175-9533-50734f403e0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:46.625825 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.625793    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f576b0a5-98fc-4175-9533-50734f403e0f-kube-api-access-sq8bh" pod "f576b0a5-98fc-4175-9533-50734f403e0f" (UID: "f576b0a5-98fc-4175-9533-50734f403e0f"). InnerVolumeSpecName "kube-api-access-sq8bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:46.625932 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.625880    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-dshm" pod "f576b0a5-98fc-4175-9533-50734f403e0f" (UID: "f576b0a5-98fc-4175-9533-50734f403e0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:46.626145 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.626125    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f576b0a5-98fc-4175-9533-50734f403e0f-tls-certs" pod "f576b0a5-98fc-4175-9533-50734f403e0f" (UID: "f576b0a5-98fc-4175-9533-50734f403e0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:31:46.640677 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.640648    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-tmp-dir" pod "f576b0a5-98fc-4175-9533-50734f403e0f" (UID: "f576b0a5-98fc-4175-9533-50734f403e0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:46.645705 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.645679    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-kserve-provision-location" pod "f576b0a5-98fc-4175-9533-50734f403e0f" (UID: "f576b0a5-98fc-4175-9533-50734f403e0f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:46.724410 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.724379    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f576b0a5-98fc-4175-9533-50734f403e0f-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.724410 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.724402    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.724410 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.724411    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.724658 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.724421    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.724658 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.724430    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.724658 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.724437    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq8bh\" (UniqueName: \"kubernetes.io/projected/f576b0a5-98fc-4175-9533-50734f403e0f-kube-api-access-sq8bh\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.724658 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:46.724445    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f576b0a5-98fc-4175-9533-50734f403e0f-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:47.557884 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:47.557843    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"
Oct 31 01:31:47.766528 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:47.766492    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"]
Oct 31 01:31:47.812799 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:47.812722    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-55567c6db7-grzct"]
Oct 31 01:31:47.944249 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:47.944214    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" path="/var/lib/kubelet/pods/f576b0a5-98fc-4175-9533-50734f403e0f/volumes"
Oct 31 01:31:52.563015 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:31:52.562972    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 31 01:32:02.563115 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:32:02.563069    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 31 01:32:12.562592 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:32:12.562544    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 31 01:32:14.152931 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:32:14.152886    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:32:22.563359 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:32:22.563313    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 31 01:32:32.563189 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:32:32.563104    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 31 01:32:42.562903 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:32:42.562863    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 31 01:32:44.152511 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:32:44.152469    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:32:52.563430 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:32:52.563391    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 31 01:33:02.563137 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:02.563099    4935 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 31 01:33:12.572818 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:12.572785    4935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:33:12.580590 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:12.580568    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:33:14.151207 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:33:14.151158    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:33:15.106829 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:15.106800    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"]
Oct 31 01:33:30.129203 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.129167    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" containerID="cri-o://5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462" gracePeriod=45
Oct 31 01:33:30.573654 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.573608    4935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 31 01:33:30.782404 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.782379    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:33:30.799541 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.799515    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-model-cache\") pod \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") "
Oct 31 01:33:30.799657 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.799547    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-dshm\") pod \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") "
Oct 31 01:33:30.799657 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.799600    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tmp-dir\") pod \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") "
Oct 31 01:33:30.799657 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.799645    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kserve-provision-location\") pod \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") "
Oct 31 01:33:30.799863 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.799714    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-home\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-home\") pod \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") "
Oct 31 01:33:30.799863 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.799743    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tls-certs\") pod \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") "
Oct 31 01:33:30.799863 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.799761    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-model-cache" pod "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" (UID: "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:30.799863 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.799812    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kube-api-access-k9nj2\" (UniqueName: \"kubernetes.io/projected/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kube-api-access-k9nj2\") pod \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") "
Oct 31 01:33:30.799863 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.799852    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-cabundle-cert\") pod \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\" (UID: \"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6\") "
Oct 31 01:33:30.800166 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.800146    4935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-model-cache\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:30.800496 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.800449    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-home" pod "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" (UID: "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:30.800865 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.800837    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-cabundle-cert" pod "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" (UID: "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:30.802360 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.802340    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-dshm" pod "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" (UID: "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:30.802523 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.802504    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tls-certs" pod "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" (UID: "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:33:30.802584 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.802565    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kube-api-access-k9nj2" pod "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" (UID: "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6"). InnerVolumeSpecName "kube-api-access-k9nj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:30.812373 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.812350    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tmp-dir" pod "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" (UID: "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:30.820999 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.820975    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kserve-provision-location" pod "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" (UID: "f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:30.901562 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.901515    4935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-dshm\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:30.901562 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.901535    4935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tmp-dir\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:30.901562 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.901544    4935 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kserve-provision-location\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:30.901562 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.901552    4935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-home\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:30.901562 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.901561    4935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-tls-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:30.901756 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.901568    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9nj2\" (UniqueName: \"kubernetes.io/projected/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-kube-api-access-k9nj2\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:30.901756 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:30.901577    4935 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6-cabundle-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:31.063876 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.063835    4935 generic.go:356] "Generic (PLEG): container finished" podID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerID="5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462" exitCode=0
Oct 31 01:33:31.064085 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.063911    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" event={"ID":"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6","Type":"ContainerDied","Data":"5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462"}
Oct 31 01:33:31.064085 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.063999    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd" event={"ID":"f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6","Type":"ContainerDied","Data":"befdb06633002c26a5f5193ef6290e407ba27a29db44b334454349ef83ca2d3d"}
Oct 31 01:33:31.064085 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.064061    4935 scope.go:122] "RemoveContainer" containerID="5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462"
Oct 31 01:33:31.064437 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.064267    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"
Oct 31 01:33:31.076007 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.075981    4935 scope.go:122] "RemoveContainer" containerID="00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef"
Oct 31 01:33:31.083757 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.083736    4935 scope.go:122] "RemoveContainer" containerID="5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462"
Oct 31 01:33:31.084114 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:33:31.084053    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462\": container with ID starting with 5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462 not found: ID does not exist" containerID="5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462"
Oct 31 01:33:31.084114 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.084081    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462"} err="failed to get container status \"5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462\": rpc error: code = NotFound desc = could not find container \"5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462\": container with ID starting with 5f42ad049cca803af00b867b40106ceb7415318a62f77a166665f90231532462 not found: ID does not exist"
Oct 31 01:33:31.084114 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.084098    4935 scope.go:122] "RemoveContainer" containerID="00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef"
Oct 31 01:33:31.084327 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:33:31.084308    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef\": container with ID starting with 00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef not found: ID does not exist" containerID="00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef"
Oct 31 01:33:31.084369 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.084334    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef"} err="failed to get container status \"00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef\": rpc error: code = NotFound desc = could not find container \"00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef\": container with ID starting with 00bb40dde49dda9712572d7e6d205c14ddcbcc333fd01057db01d0c16e7cd4ef not found: ID does not exist"
Oct 31 01:33:31.223903 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.223845    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"]
Oct 31 01:33:31.229949 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.229929    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-z6qqd"]
Oct 31 01:33:31.943355 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:31.943324    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" path="/var/lib/kubelet/pods/f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6/volumes"
Oct 31 01:33:44.150539 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:33:44.150504    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:33:48.327374 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:48.327345    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:33:48.344398 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:33:48.344377    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pcbjl_c44bba77-c0b2-49cc-92c1-836cf8013950/console-operator/2.log"
Oct 31 01:34:14.151062 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:34:14.151011    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:34:43.093834 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.093802    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-whh42/must-gather-w7rbz"]
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094175    4935 cpu_manager.go:414] "containerMap: removing container" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="storage-initializer"
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094188    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="storage-initializer"
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094208    4935 cpu_manager.go:414] "containerMap: removing container" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy"
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094213    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy"
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094225    4935 cpu_manager.go:414] "containerMap: removing container" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main"
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094231    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main"
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094238    4935 cpu_manager.go:414] "containerMap: removing container" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="storage-initializer"
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094243    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="storage-initializer"
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094255    4935 cpu_manager.go:414] "containerMap: removing container" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy"
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094260    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy"
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094268    4935 cpu_manager.go:414] "containerMap: removing container" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main"
Oct 31 01:34:43.094298 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094272    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main"
Oct 31 01:34:43.094700 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094325    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy"
Oct 31 01:34:43.094700 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094333    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f576b0a5-98fc-4175-9533-50734f403e0f" containerName="main"
Oct 31 01:34:43.094700 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094341    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="82142e5f-4d6e-4feb-be27-157d733bef6f" containerName="istio-proxy"
Oct 31 01:34:43.094700 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.094346    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35cc598-1b4e-4e75-a7c7-f0cb70b2d8b6" containerName="main"
Oct 31 01:34:43.097541 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.097522    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:34:43.137726 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.137701    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fc04d64-8cdc-4999-918e-bc59e486f3f2-must-gather-output\") pod \"must-gather-w7rbz\" (UID: \"0fc04d64-8cdc-4999-918e-bc59e486f3f2\") " pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:34:43.137824 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.137764    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6lpl\" (UniqueName: \"kubernetes.io/projected/0fc04d64-8cdc-4999-918e-bc59e486f3f2-kube-api-access-m6lpl\") pod \"must-gather-w7rbz\" (UID: \"0fc04d64-8cdc-4999-918e-bc59e486f3f2\") " pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:34:43.151891 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.151864    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-whh42/must-gather-w7rbz"]
Oct 31 01:34:43.196380 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.196355    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-whh42\"/\"openshift-service-ca.crt\""
Oct 31 01:34:43.196380 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.196368    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-whh42\"/\"default-dockercfg-mqgfl\""
Oct 31 01:34:43.196508 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.196396    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-whh42\"/\"kube-root-ca.crt\""
Oct 31 01:34:43.238487 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.238463    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fc04d64-8cdc-4999-918e-bc59e486f3f2-must-gather-output\") pod \"must-gather-w7rbz\" (UID: \"0fc04d64-8cdc-4999-918e-bc59e486f3f2\") " pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:34:43.238582 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.238515    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6lpl\" (UniqueName: \"kubernetes.io/projected/0fc04d64-8cdc-4999-918e-bc59e486f3f2-kube-api-access-m6lpl\") pod \"must-gather-w7rbz\" (UID: \"0fc04d64-8cdc-4999-918e-bc59e486f3f2\") " pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:34:43.238760 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.238744    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fc04d64-8cdc-4999-918e-bc59e486f3f2-must-gather-output\") pod \"must-gather-w7rbz\" (UID: \"0fc04d64-8cdc-4999-918e-bc59e486f3f2\") " pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:34:43.300339 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.300315    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6lpl\" (UniqueName: \"kubernetes.io/projected/0fc04d64-8cdc-4999-918e-bc59e486f3f2-kube-api-access-m6lpl\") pod \"must-gather-w7rbz\" (UID: \"0fc04d64-8cdc-4999-918e-bc59e486f3f2\") " pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:34:43.410960 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.410915    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:34:43.786083 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.786054    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-whh42/must-gather-w7rbz"]
Oct 31 01:34:43.793494 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:34:43.793465    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fc04d64_8cdc_4999_918e_bc59e486f3f2.slice/crio-83595d353e4bab6370db4d53b85f0680de7d5ec662926e5376dbed3eee247197 WatchSource:0}: Error finding container 83595d353e4bab6370db4d53b85f0680de7d5ec662926e5376dbed3eee247197: Status 404 returned error can't find the container with id 83595d353e4bab6370db4d53b85f0680de7d5ec662926e5376dbed3eee247197
Oct 31 01:34:43.795290 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:43.795266    4935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:34:44.151376 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:34:44.151305    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:34:44.375660 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:44.375625    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-whh42/must-gather-w7rbz" event={"ID":"0fc04d64-8cdc-4999-918e-bc59e486f3f2","Type":"ContainerStarted","Data":"83595d353e4bab6370db4d53b85f0680de7d5ec662926e5376dbed3eee247197"}
Oct 31 01:34:52.422080 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:52.421978    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-whh42/must-gather-w7rbz" event={"ID":"0fc04d64-8cdc-4999-918e-bc59e486f3f2","Type":"ContainerStarted","Data":"bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d"}
Oct 31 01:34:52.422080 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:52.422027    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-whh42/must-gather-w7rbz" event={"ID":"0fc04d64-8cdc-4999-918e-bc59e486f3f2","Type":"ContainerStarted","Data":"b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149"}
Oct 31 01:34:52.532411 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:34:52.532358    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-whh42/must-gather-w7rbz" podStartSLOduration=1.517900024 podStartE2EDuration="9.53234548s" totalImagesPullingTime="8.014445456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:52.531657155 +0000 UTC m=+3964.714186451" watchObservedRunningTime="2026-10-31 01:34:52.53234548 +0000 UTC m=+3964.714874773"
Oct 31 01:35:07.689173 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:07.689133    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"]
Oct 31 01:35:07.689557 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:07.689408    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" podUID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" containerName="istio-proxy" containerID="cri-o://b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc" gracePeriod=30
Oct 31 01:35:12.936476 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:12.936448    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:35:13.000399 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.000364    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-token\" (UniqueName: \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-token\") pod \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") "
Oct 31 01:35:13.000542 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.000462    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-credential-socket\") pod \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") "
Oct 31 01:35:13.000677 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.000651    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-data\") pod \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") "
Oct 31 01:35:13.000812 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.000710    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-envoy\") pod \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") "
Oct 31 01:35:13.000812 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.000745    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-kube-api-access-skbwp\" (UniqueName: \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-kube-api-access-skbwp\") pod \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") "
Oct 31 01:35:13.000812 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.000797    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-podinfo\") pod \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") "
Oct 31 01:35:13.001003 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.000812    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-credential-socket" pod "bf49fc49-8332-42c1-9ce5-e62fd20fc666" (UID: "bf49fc49-8332-42c1-9ce5-e62fd20fc666"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:13.001003 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.000854    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-certs\") pod \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") "
Oct 31 01:35:13.001003 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.000902    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-socket\") pod \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") "
Oct 31 01:35:13.001003 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.000929    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istiod-ca-cert\") pod \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\" (UID: \"bf49fc49-8332-42c1-9ce5-e62fd20fc666\") "
Oct 31 01:35:13.001311 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.000981    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-data" pod "bf49fc49-8332-42c1-9ce5-e62fd20fc666" (UID: "bf49fc49-8332-42c1-9ce5-e62fd20fc666"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:13.001311 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.001214    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-certs" pod "bf49fc49-8332-42c1-9ce5-e62fd20fc666" (UID: "bf49fc49-8332-42c1-9ce5-e62fd20fc666"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:13.001311 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.001290    4935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-credential-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:35:13.001311 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.001311    4935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-data\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:35:13.001546 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.001326    4935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-certs\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:35:13.001885 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.001832    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-socket" pod "bf49fc49-8332-42c1-9ce5-e62fd20fc666" (UID: "bf49fc49-8332-42c1-9ce5-e62fd20fc666"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:13.002305 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.002276    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istiod-ca-cert" pod "bf49fc49-8332-42c1-9ce5-e62fd20fc666" (UID: "bf49fc49-8332-42c1-9ce5-e62fd20fc666"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:35:13.003253 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.003231    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-token" pod "bf49fc49-8332-42c1-9ce5-e62fd20fc666" (UID: "bf49fc49-8332-42c1-9ce5-e62fd20fc666"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:13.003407 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.003384    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-envoy" pod "bf49fc49-8332-42c1-9ce5-e62fd20fc666" (UID: "bf49fc49-8332-42c1-9ce5-e62fd20fc666"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:13.003512 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.003498    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-podinfo" pod "bf49fc49-8332-42c1-9ce5-e62fd20fc666" (UID: "bf49fc49-8332-42c1-9ce5-e62fd20fc666"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:35:13.003919 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.003894    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-kube-api-access-skbwp" pod "bf49fc49-8332-42c1-9ce5-e62fd20fc666" (UID: "bf49fc49-8332-42c1-9ce5-e62fd20fc666"). InnerVolumeSpecName "kube-api-access-skbwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:13.105068 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.101918    4935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-envoy\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:35:13.105068 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.101951    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skbwp\" (UniqueName: \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-kube-api-access-skbwp\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:35:13.105068 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.101966    4935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-podinfo\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:35:13.105068 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.101978    4935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49fc49-8332-42c1-9ce5-e62fd20fc666-workload-socket\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:35:13.105068 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.101990    4935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istiod-ca-cert\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:35:13.105068 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.102003    4935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf49fc49-8332-42c1-9ce5-e62fd20fc666-istio-token\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:35:13.522187 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.522150    4935 generic.go:356] "Generic (PLEG): container finished" podID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" containerID="b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc" exitCode=0
Oct 31 01:35:13.522370 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.522217    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"
Oct 31 01:35:13.522370 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.522235    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" event={"ID":"bf49fc49-8332-42c1-9ce5-e62fd20fc666","Type":"ContainerDied","Data":"b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc"}
Oct 31 01:35:13.522370 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.522312    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w" event={"ID":"bf49fc49-8332-42c1-9ce5-e62fd20fc666","Type":"ContainerDied","Data":"44fc6243c9725a874f281719e3fffe94ae50e42509269ae97824137b101fd3b9"}
Oct 31 01:35:13.522370 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.522334    4935 scope.go:122] "RemoveContainer" containerID="b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc"
Oct 31 01:35:13.531603 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.531586    4935 scope.go:122] "RemoveContainer" containerID="b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc"
Oct 31 01:35:13.531853 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:35:13.531836    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc\": container with ID starting with b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc not found: ID does not exist" containerID="b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc"
Oct 31 01:35:13.531901 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.531860    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc"} err="failed to get container status \"b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc\": rpc error: code = NotFound desc = could not find container \"b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc\": container with ID starting with b47d0b67410eec98f630ba696e98aee4477e0293512dbe9cb4a7adcb84eb6ddc not found: ID does not exist"
Oct 31 01:35:13.680535 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.680512    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"]
Oct 31 01:35:13.726874 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.726853    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-kb28w"]
Oct 31 01:35:13.943241 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:13.943168    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" path="/var/lib/kubelet/pods/bf49fc49-8332-42c1-9ce5-e62fd20fc666/volumes"
Oct 31 01:35:14.152836 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:35:14.152799    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:35:44.151561 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:35:44.151524    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:35:48.674299 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:48.674271    4935 scope.go:122] "RemoveContainer" containerID="19b98bfd674abcc78458e790829da247268edb24735753a7e0202c3d27d6f1a4"
Oct 31 01:35:48.683672 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:48.683652    4935 scope.go:122] "RemoveContainer" containerID="9954eb1910806f0e8d860770dcd55691f2444f632e5b6421e32fb278d1fe8ca0"
Oct 31 01:35:55.642925 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:55.642895    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-568c495856-q9tdb_42e950da-255b-405e-9bf0-621c4f39c959/istio-proxy/0.log"
Oct 31 01:35:56.190862 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:56.190826    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-568c495856-vgwht_b36efdef-c7ef-4c3e-a887-05961228c7bb/istio-proxy/0.log"
Oct 31 01:35:56.514002 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:35:56.513980    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55c864699c-n4cfn_2e209316-e440-4bdf-b2d9-cc79ac3e670f/router/0.log"
Oct 31 01:36:00.314806 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:00.314775    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-568c495856-q9tdb_42e950da-255b-405e-9bf0-621c4f39c959/istio-proxy/0.log"
Oct 31 01:36:00.744342 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:00.744307    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-568c495856-vgwht_b36efdef-c7ef-4c3e-a887-05961228c7bb/istio-proxy/0.log"
Oct 31 01:36:01.067541 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:01.067468    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55c864699c-n4cfn_2e209316-e440-4bdf-b2d9-cc79ac3e670f/router/0.log"
Oct 31 01:36:04.623270 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:04.623241    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-68df894cb4-hn7t7_a17d7101-2b60-4959-bb10-dcc468e05eb4/manager/0.log"
Oct 31 01:36:04.831995 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:04.831972    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-n9j29_f2f4fcd6-6bde-4ad9-84e0-6519fc08848d/kuadrant-console-plugin/0.log"
Oct 31 01:36:07.770445 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:07.770382    4935 generic.go:356] "Generic (PLEG): container finished" podID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" containerID="b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149" exitCode=0
Oct 31 01:36:07.770802 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:07.770449    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-whh42/must-gather-w7rbz" event={"ID":"0fc04d64-8cdc-4999-918e-bc59e486f3f2","Type":"ContainerDied","Data":"b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149"}
Oct 31 01:36:07.770802 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:07.770739    4935 scope.go:122] "RemoveContainer" containerID="b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149"
Oct 31 01:36:08.629285 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:08.629261    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-whh42_must-gather-w7rbz_0fc04d64-8cdc-4999-918e-bc59e486f3f2/gather/0.log"
Oct 31 01:36:14.152061 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:36:14.152011    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:36:15.249966 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.249929    4935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-whh42/must-gather-w7rbz"]
Oct 31 01:36:15.250440 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.250174    4935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-must-gather-whh42/must-gather-w7rbz" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" containerName="copy" containerID="cri-o://bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d" gracePeriod=2
Oct 31 01:36:15.296596 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.296565    4935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-whh42/must-gather-w7rbz"]
Oct 31 01:36:15.298009 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:36:15.297976    4935 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-w7rbz\" is forbidden: User \"system:node:ip-10-0-132-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-whh42\": no relationship found between node 'ip-10-0-132-125.us-west-2.compute.internal' and this object" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:36:15.533732 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.533710    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-whh42_must-gather-w7rbz_0fc04d64-8cdc-4999-918e-bc59e486f3f2/copy/0.log"
Oct 31 01:36:15.536373 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.536355    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:36:15.584288 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:36:15.584258    4935 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-w7rbz\" is forbidden: User \"system:node:ip-10-0-132-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-whh42\": no relationship found between node 'ip-10-0-132-125.us-west-2.compute.internal' and this object" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:36:15.623919 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.623895    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc04d64-8cdc-4999-918e-bc59e486f3f2-kube-api-access-m6lpl\" (UniqueName: \"kubernetes.io/projected/0fc04d64-8cdc-4999-918e-bc59e486f3f2-kube-api-access-m6lpl\") pod \"0fc04d64-8cdc-4999-918e-bc59e486f3f2\" (UID: \"0fc04d64-8cdc-4999-918e-bc59e486f3f2\") "
Oct 31 01:36:15.624068 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.623972    4935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc04d64-8cdc-4999-918e-bc59e486f3f2-must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fc04d64-8cdc-4999-918e-bc59e486f3f2-must-gather-output\") pod \"0fc04d64-8cdc-4999-918e-bc59e486f3f2\" (UID: \"0fc04d64-8cdc-4999-918e-bc59e486f3f2\") "
Oct 31 01:36:15.626206 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.626180    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc04d64-8cdc-4999-918e-bc59e486f3f2-kube-api-access-m6lpl" pod "0fc04d64-8cdc-4999-918e-bc59e486f3f2" (UID: "0fc04d64-8cdc-4999-918e-bc59e486f3f2"). InnerVolumeSpecName "kube-api-access-m6lpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:15.632975 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.632948    4935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc04d64-8cdc-4999-918e-bc59e486f3f2-must-gather-output" pod "0fc04d64-8cdc-4999-918e-bc59e486f3f2" (UID: "0fc04d64-8cdc-4999-918e-bc59e486f3f2"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:15.724848 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.724823    4935 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fc04d64-8cdc-4999-918e-bc59e486f3f2-must-gather-output\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:36:15.724848 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.724847    4935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6lpl\" (UniqueName: \"kubernetes.io/projected/0fc04d64-8cdc-4999-918e-bc59e486f3f2-kube-api-access-m6lpl\") on node \"ip-10-0-132-125.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:36:15.804431 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.804379    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-whh42_must-gather-w7rbz_0fc04d64-8cdc-4999-918e-bc59e486f3f2/copy/0.log"
Oct 31 01:36:15.804738 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.804715    4935 generic.go:356] "Generic (PLEG): container finished" podID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" containerID="bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d" exitCode=143
Oct 31 01:36:15.804807 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.804757    4935 scope.go:122] "RemoveContainer" containerID="bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d"
Oct 31 01:36:15.804807 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.804760    4935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:36:15.812633 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.812610    4935 scope.go:122] "RemoveContainer" containerID="b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149"
Oct 31 01:36:15.825186 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.825169    4935 scope.go:122] "RemoveContainer" containerID="bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d"
Oct 31 01:36:15.825448 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:36:15.825432    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d\": container with ID starting with bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d not found: ID does not exist" containerID="bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d"
Oct 31 01:36:15.825524 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.825459    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d"} err="failed to get container status \"bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d\": rpc error: code = NotFound desc = could not find container \"bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d\": container with ID starting with bcb3826e7eb63e66ee53e322f29aac53f86186d3823b83a5def962a83abaf76d not found: ID does not exist"
Oct 31 01:36:15.825524 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.825482    4935 scope.go:122] "RemoveContainer" containerID="b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149"
Oct 31 01:36:15.825713 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:36:15.825694    4935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149\": container with ID starting with b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149 not found: ID does not exist" containerID="b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149"
Oct 31 01:36:15.825792 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.825720    4935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149"} err="failed to get container status \"b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149\": rpc error: code = NotFound desc = could not find container \"b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149\": container with ID starting with b4885dac828dde8155df5ba8b949ed0b571d06e8d06e8017b212307699c24149 not found: ID does not exist"
Oct 31 01:36:15.852754 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:36:15.852726    4935 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-w7rbz\" is forbidden: User \"system:node:ip-10-0-132-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-whh42\": no relationship found between node 'ip-10-0-132-125.us-west-2.compute.internal' and this object" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:36:15.900408 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:36:15.900382    4935 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-w7rbz\" is forbidden: User \"system:node:ip-10-0-132-125.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-whh42\": no relationship found between node 'ip-10-0-132-125.us-west-2.compute.internal' and this object" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" pod="openshift-must-gather-whh42/must-gather-w7rbz"
Oct 31 01:36:15.942982 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:15.942957    4935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" path="/var/lib/kubelet/pods/0fc04d64-8cdc-4999-918e-bc59e486f3f2/volumes"
Oct 31 01:36:20.847384 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:20.847355    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wdzhr_261b0474-eb3d-4f3c-b33d-01856306e928/global-pull-secret-syncer/0.log"
Oct 31 01:36:21.323453 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:21.323426    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8tt4s_408faab9-57a4-4c18-86ae-68772011de21/connection-checker/0.log"
Oct 31 01:36:21.754827 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:21.754800    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hvfpm_dc22e5f2-92a8-424b-a3a0-c2f99a43b9ed/connection-checker/0.log"
Oct 31 01:36:22.196431 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:22.196365    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tkpbr_81be0f35-6496-46d5-a27b-46bd185e3993/connection-checker/0.log"
Oct 31 01:36:23.576682 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:23.576653    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l64l2_7ab38846-5d17-4bf1-92c2-ac823690fc98/konnectivity-agent/0.log"
Oct 31 01:36:24.394015 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:24.393983    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-125.us-west-2.compute.internal_859c761ccba0302346f6d0c201b0e803/haproxy/0.log"
Oct 31 01:36:30.032439 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:30.032406    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-68df894cb4-hn7t7_a17d7101-2b60-4959-bb10-dcc468e05eb4/manager/0.log"
Oct 31 01:36:30.445319 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:30.445256    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-n9j29_f2f4fcd6-6bde-4ad9-84e0-6519fc08848d/kuadrant-console-plugin/0.log"
Oct 31 01:36:32.722445 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.722414    4935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"]
Oct 31 01:36:32.722814 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.722758    4935 cpu_manager.go:414] "containerMap: removing container" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" containerName="gather"
Oct 31 01:36:32.722814 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.722769    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" containerName="gather"
Oct 31 01:36:32.722814 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.722781    4935 cpu_manager.go:414] "containerMap: removing container" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" containerName="copy"
Oct 31 01:36:32.722814 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.722787    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" containerName="copy"
Oct 31 01:36:32.722814 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.722817    4935 cpu_manager.go:414] "containerMap: removing container" podUID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" containerName="istio-proxy"
Oct 31 01:36:32.723060 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.722823    4935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" containerName="istio-proxy"
Oct 31 01:36:32.723060 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.722881    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf49fc49-8332-42c1-9ce5-e62fd20fc666" containerName="istio-proxy"
Oct 31 01:36:32.723060 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.722891    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" containerName="gather"
Oct 31 01:36:32.723060 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.722898    4935 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc04d64-8cdc-4999-918e-bc59e486f3f2" containerName="copy"
Oct 31 01:36:32.725892 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.725873    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.779620 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.779589    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjbw9\"/\"openshift-service-ca.crt\""
Oct 31 01:36:32.780409 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.780387    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"]
Oct 31 01:36:32.825453 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.825416    4935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjbw9\"/\"kube-root-ca.crt\""
Oct 31 01:36:32.825550 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.825489    4935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sjbw9\"/\"default-dockercfg-jfw5t\""
Oct 31 01:36:32.853508 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.853487    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-podres\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.853628 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.853516    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-lib-modules\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.853628 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.853535    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-proc\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.853628 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.853606    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-sys\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.853771 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.853658    4935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw9q2\" (UniqueName: \"kubernetes.io/projected/73520445-a129-4ded-930b-d12f0e3daf2c-kube-api-access-pw9q2\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.954425 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.954398    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-podres\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.954555 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.954439    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-lib-modules\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.954555 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.954481    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-proc\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.954555 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.954519    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-sys\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.954555 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.954523    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-podres\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.954555 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.954543    4935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw9q2\" (UniqueName: \"kubernetes.io/projected/73520445-a129-4ded-930b-d12f0e3daf2c-kube-api-access-pw9q2\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.954787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.954597    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-proc\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.954787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.954604    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-sys\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:32.954787 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:32.954608    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73520445-a129-4ded-930b-d12f0e3daf2c-lib-modules\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:33.013969 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:33.013944    4935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw9q2\" (UniqueName: \"kubernetes.io/projected/73520445-a129-4ded-930b-d12f0e3daf2c-kube-api-access-pw9q2\") pod \"perf-node-gather-daemonset-khq7t\" (UID: \"73520445-a129-4ded-930b-d12f0e3daf2c\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:33.037870 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:33.037845    4935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:33.407335 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:33.407269    4935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"]
Oct 31 01:36:33.414027 ip-10-0-132-125 kubenswrapper[4935]: W1031 01:36:33.413996    4935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod73520445_a129_4ded_930b_d12f0e3daf2c.slice/crio-75a89907fb59f963ff50caeafe24b38d61ba534c64eb4521ed8489e9d5c4da95 WatchSource:0}: Error finding container 75a89907fb59f963ff50caeafe24b38d61ba534c64eb4521ed8489e9d5c4da95: Status 404 returned error can't find the container with id 75a89907fb59f963ff50caeafe24b38d61ba534c64eb4521ed8489e9d5c4da95
Oct 31 01:36:33.883940 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:33.883904    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t" event={"ID":"73520445-a129-4ded-930b-d12f0e3daf2c","Type":"ContainerStarted","Data":"815c7dc393f2fac877d581a280be575762513c1579dacd555a2c6dd9dcd4aff9"}
Oct 31 01:36:33.884327 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:33.883959    4935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t" event={"ID":"73520445-a129-4ded-930b-d12f0e3daf2c","Type":"ContainerStarted","Data":"75a89907fb59f963ff50caeafe24b38d61ba534c64eb4521ed8489e9d5c4da95"}
Oct 31 01:36:33.884327 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:33.884072    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:33.995018 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:33.994978    4935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t" podStartSLOduration=1.994964755 podStartE2EDuration="1.994964755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:33.994360736 +0000 UTC m=+4066.176890030" watchObservedRunningTime="2026-10-31 01:36:33.994964755 +0000 UTC m=+4066.177494045"
Oct 31 01:36:37.090988 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:37.090956    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-r9hjn_54ecc909-648d-4848-9f49-464285458129/cluster-monitoring-operator/0.log"
Oct 31 01:36:39.898766 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:39.898740    4935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-khq7t"
Oct 31 01:36:40.778952 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:40.778920    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bn9lb_3318c24a-33bb-481b-8e63-0a4d5f032b93/node-exporter/0.log"
Oct 31 01:36:41.190552 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:41.190479    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bn9lb_3318c24a-33bb-481b-8e63-0a4d5f032b93/kube-rbac-proxy/0.log"
Oct 31 01:36:41.600016 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:41.599996    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bn9lb_3318c24a-33bb-481b-8e63-0a4d5f032b93/init-textfile/0.log"
Oct 31 01:36:44.151700 ip-10-0-132-125 kubenswrapper[4935]: E1031 01:36:44.151664    4935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x7l7z" containerName="extractor"
Oct 31 01:36:48.223621 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:48.223590    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-7hw5x_420c4e76-1007-4c7b-80b3-441ba4fdcf4a/prometheus-operator-admission-webhook/0.log"
Oct 31 01:36:54.731681 ip-10-0-132-125 kubenswrapper[4935]: I1031 01:36:54.731650    4935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-9lmll_d543da15-0c67-4912-a39e-04235a6afa20/networking-console-plugin/0.log"